[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483044173 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002379] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007437] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.011114] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.015054] *** VALIDATE selinux *** [ 0.024372] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029498] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032120] *** VALIDATE tmpfs *** [ 0.033438] *** VALIDATE proc *** [ 0.034233] *** VALIDATE cgroup *** [ 0.036006] *** VALIDATE cgroup2 *** [ 0.037273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039087] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041029] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.045189] debug: unmapping init [mem 0xffffffffac659000-0xffffffffac660fff] [ 0.047179] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048640] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053013] ... max period: 00007fffffffffff [ 0.054018] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056315] rcu: Hierarchical SRCU implementation. [ 0.058358] smp: Bringing up secondary CPUs ... [ 0.059579] x86: Booting SMP configuration: [ 0.060022] .... node #0, CPUs: #1 #2 #3 [ 0.064283] smp: Brought up 1 node, 4 CPUs [ 0.066017] smpboot: Max logical packages: 1 [ 0.067018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.181240] node 0 deferred pages initialised in 111ms [ 0.183133] devtmpfs: initialized [ 0.185326] x86/mm: Memory block size: 128MB [ 0.187745] gcov: version magic: 0x41383552 [ 0.191381] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.192105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.193362] pinctrl core: initialized pinctrl subsystem [ 0.194279] [ 0.194930] ************************************************************* [ 0.195024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196019] ** ** [ 0.197020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.198020] ** ** [ 0.199019] ** This means that this kernel is built to expose internal ** [ 0.200019] ** IOMMU data structures, which may compromise security on ** [ 0.201020] ** your system. ** [ 0.202016] ** ** [ 0.203018] ** If you see this message and you are not debugging the ** [ 0.204018] ** kernel, report this immediately to your vendor! ** [ 0.205014] ** ** [ 0.206016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207015] ************************************************************* [ 0.208799] NET: Registered protocol family 16 [ 0.209526] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212561] cpuidle: using governor menu [ 0.214637] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216428] PCI: Using configuration type 1 for base access [ 0.219125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231023] cryptd: max_cpu_qlen set to 1000 [ 0.233237] ACPI: Added _OSI(Module Device) [ 0.234015] ACPI: Added _OSI(Processor Device) [ 0.235015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.236017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240123] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248077] ACPI: Interpreter enabled [ 0.249068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250000] ACPI: Using IOAPIC for interrupt routing [ 0.252101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255417] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265878] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279827] acpiphp: Slot [2] registered [ 0.281225] acpiphp: Slot [5] registered [ 0.283176] acpiphp: Slot [6] registered [ 0.285131] acpiphp: Slot [7] registered [ 0.286118] acpiphp: Slot [8] registered [ 0.288159] acpiphp: Slot [9] registered [ 0.289144] acpiphp: Slot [10] registered [ 0.291107] acpiphp: Slot [3] registered [ 0.292127] acpiphp: Slot [4] registered [ 0.293082] acpiphp: Slot [11] registered [ 0.295115] acpiphp: Slot [12] registered [ 0.297103] acpiphp: Slot [13] registered [ 0.298086] acpiphp: Slot [14] registered [ 0.300085] acpiphp: Slot [15] registered [ 0.301119] acpiphp: Slot [16] registered [ 0.302104] acpiphp: Slot [17] registered [ 0.304085] acpiphp: Slot [18] registered [ 0.305101] acpiphp: Slot [19] registered [ 0.307087] acpiphp: Slot [20] registered [ 0.309107] acpiphp: Slot [21] registered [ 0.310108] acpiphp: Slot [22] registered [ 0.311133] acpiphp: Slot [23] registered [ 0.313096] acpiphp: Slot [24] registered [ 0.315082] acpiphp: Slot [25] registered [ 0.316087] acpiphp: Slot [26] registered [ 0.317086] acpiphp: Slot [27] registered [ 0.319102] acpiphp: Slot [28] registered [ 0.320108] acpiphp: Slot [29] registered [ 0.322094] acpiphp: Slot [30] registered [ 0.323080] acpiphp: Slot [31] registered [ 0.324074] PCI host bridge to bus 0000:00 [ 0.326017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339213] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341544] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345531] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371036] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376119] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380065] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387774] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.396016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422401] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.437020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.446022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.466023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.478264] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.497015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.508017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.528023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.540265] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.547015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.553015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.578867] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.584020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.590017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.602017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.613043] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.621023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.631023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.654024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.664000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.677014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.686014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.713015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.728250] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.731521] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.735107] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.741363] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.746305] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.753037] iommu: Default domain type: Passthrough [ 0.755452] SCSI subsystem initialized [ 0.756116] ACPI: bus type USB registered [ 0.758128] usbcore: registered new interface driver usbfs [ 0.761108] usbcore: registered new interface driver hub [ 0.762101] usbcore: registered new device driver usb [ 0.765232] pps_core: LinuxPPS API ver. 1 registered [ 0.767015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.771111] PTP clock support registered [ 0.774098] EDAC MC: Ver: 3.0.0 [ 0.776172] PCI: Using ACPI for IRQ routing [ 0.779626] NetLabel: Initializing [ 0.781022] NetLabel: domain hash size = 128 [ 0.783019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.785135] NetLabel: unlabeled traffic allowed by default [ 0.788202] vgaarb: loaded [ 0.790263] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.791016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.799000] clocksource: Switched to clocksource kvm-clock [ 0.907034] VFS: Disk quotas dquot_6.6.0 [ 0.908366] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.910349] *** VALIDATE ramfs *** [ 0.911453] *** VALIDATE hugetlbfs *** [ 0.912734] pnp: PnP ACPI init [ 0.915546] pnp: PnP ACPI: found 6 devices [ 0.931048] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.933981] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935897] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.937912] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.940115] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.942312] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.944850] NET: Registered protocol family 2 [ 0.947286] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.952034] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.955588] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.961483] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.965252] TCP: Hash tables configured (established 65536 bind 65536) [ 0.968623] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.972442] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975049] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978058] NET: Registered protocol family 1 [ 0.981095] RPC: Registered named UNIX socket transport module. [ 0.983345] RPC: Registered udp transport module. [ 0.985078] RPC: Registered tcp transport module. [ 0.987329] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.989970] NET: Registered protocol family 44 [ 0.991865] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.994228] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.996591] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.999804] PCI: CLS 0 bytes, default 64 [ 1.001573] Unpacking initramfs... [ 2.470401] debug: unmapping init [mem 0xffff8f4bbcc54000-0xffff8f4bbffbffff] [ 2.477291] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.479971] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.484725] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.981587] Initialise system trusted keyrings [ 2.983559] Key type blacklist registered [ 2.985763] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.998705] zbud: loaded [ 3.002447] *** VALIDATE nfs *** [ 3.005468] *** VALIDATE nfs4 *** [ 3.007237] pstore: using deflate compression [ 3.011076] Platform Keyring initialized [ 3.113748] NET: Registered protocol family 38 [ 3.115720] Key type asymmetric registered [ 3.117641] Asymmetric key parser 'x509' registered [ 3.119742] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.123597] io scheduler mq-deadline registered [ 3.125568] io scheduler kyber registered [ 3.127429] io scheduler bfq registered [ 3.129684] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.133210] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.136409] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.139614] ACPI: Power Button [PWRF] [ 3.145126] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.151929] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.165411] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.177785] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.192507] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.222120] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.251868] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.257236] Non-volatile memory driver v1.3 [ 3.259236] Linux agpgart interface v0.103 [ 3.294026] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.297082] vda: detected capacity change from 0 to 75100160 [ 3.310898] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.313916] vdb: detected capacity change from 0 to 1073741824 [ 3.335335] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.338021] vdc: detected capacity change from 0 to 2621440000 [ 3.353450] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.356225] vdd: detected capacity change from 0 to 2621440000 [ 3.373730] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.376617] vde: detected capacity change from 0 to 4294967296 [ 3.393571] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.396447] vdf: detected capacity change from 0 to 4294967296 [ 3.411683] libphy: Fixed MDIO Bus: probed [ 3.417303] usbcore: registered new interface driver usbserial_generic [ 3.419908] usbserial: USB Serial support registered for generic [ 3.422770] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.427730] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.429884] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.432408] mousedev: PS/2 mouse device common for all mice [ 3.436377] rtc_cmos 00:05: RTC can wake from S4 [ 3.439519] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.443256] rtc_cmos 00:05: registered as rtc0 [ 3.446839] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.448447] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.451259] intel_pstate: CPU model not supported [ 3.455425] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.461368] hid: raw HID events driver (C) Jiri Kosina [ 3.463533] usbcore: registered new interface driver usbhid [ 3.465546] usbhid: USB HID core driver [ 3.467238] drop_monitor: Initializing network drop monitor service [ 3.469954] Initializing XFRM netlink socket [ 3.472069] NET: Registered protocol family 10 [ 3.475424] Segment Routing with IPv6 [ 3.477214] NET: Registered protocol family 17 [ 3.479662] mpls_gso: MPLS GSO support [ 3.486504] RAS: Correctable Errors collector initialized. [ 3.489431] AVX version of gcm_enc/dec engaged. [ 3.491652] AES CTR mode by8 optimization enabled [ 3.577289] sched_clock: Marking stable (3577250290, 0)->(4513051554, -935801264) [ 3.580575] registered taskstats version 1 [ 3.583195] Loading compiled-in X.509 certificates [ 3.585193] zswap: loaded using pool lzo/zbud [ 3.609601] Key type big_key registered [ 3.622802] Key type encrypted registered [ 3.624865] ima: No TPM chip found, activating TPM-bypass! [ 3.627400] ima: Allocated hash algorithm: sha1 [ 3.629419] ima: No architecture policies found [ 3.631514] evm: Initialising EVM extended attributes: [ 3.633590] evm: security.selinux [ 3.635095] evm: security.ima [ 3.636582] evm: security.capability [ 3.638119] evm: HMAC attrs: 0x1 [ 3.640799] rtc_cmos 00:05: setting system clock to 2026-08-27 10:22:37 UTC (1787826157) [ 3.647769] debug: unmapping init [mem 0xffffffffad603000-0xffffffffad7fffff] [ 3.651540] debug: unmapping init [mem 0xffffffffac382000-0xffffffffac658fff] [ 3.658118] Write protecting the kernel read-only data: 28672k [ 3.661328] debug: unmapping init [mem 0xffffffffaaa03000-0xffffffffaabfffff] [ 3.664106] debug: unmapping init [mem 0xffffffffab314000-0xffffffffab3fffff] [ 3.698350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.708042] systemd[1]: Detected virtualization kvm. [ 3.710446] systemd[1]: Detected architecture x86-64. [ 3.712699] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.744316] systemd[1]: No hostname configured. [ 3.746157] systemd[1]: Set hostname to . [ 3.748577] random: systemd: uninitialized urandom read (16 bytes read) [ 3.751299] systemd[1]: Initializing machine ID from random generator. [ 3.896608] random: systemd: uninitialized urandom read (16 bytes read) [ 3.899265] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.904436] random: systemd: uninitialized urandom read (16 bytes read) [ 3.907605] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.912962] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.563693] device-mapper: uevent: version 1.0.3 [ 4.566097] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.347848] virtio_net virtio0 ens2: renamed from eth0 [ 5.362598] random: fast init done [ 5.394894] scsi host0: ata_piix [ 5.426770] scsi host1: ata_piix [ 5.428537] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.431418] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.970173] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.143671] random: crng init done [ 10.147556] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.678543] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.836403] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.102700] SELinux: Disabled at runtime. [ 12.165337] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.174541] systemd[1]: Detected virtualization kvm. [ 12.176662] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.681990] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.685709] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.690767] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.695020] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.698366] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.705724] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.710936] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ 12.882197] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.161808] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.449595] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.465145] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.602966] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.614609] EDAC sbridge: Ver: 1.1.2 [ 15.301221] Key type dns_resolver registered [ 15.608365] NFS: Registering the id_resolver key type [ 15.610278] Key type id_resolver registered [ 15.611926] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg633-server login: [ 48.215070] libcfs: loading out-of-tree module taints kernel. [ 48.266262] Key type ._llcrypt registered [ 48.273187] Key type .llcrypt registered [ 48.374936] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 67.345958] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 68.868771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.879029] alg: No test for adler32 (adler32-zlib) [ 70.226335] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 71.060689] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 72.833228] Key type lgssc registered [ 74.522843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.447721] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.795501] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 141.941405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.980611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.218133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.239794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.324709] Lustre: lustre-MDT0000: new disk, initializing [ 143.458076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.478569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.864631] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.580378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.707681] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.739623] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.743786] Lustre: Skipped 1 previous similar message [ 162.865599] Lustre: lustre-MDT0001: new disk, initializing [ 162.964754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.058797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.080262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.134753] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.571519] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 183.993665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.242219] Lustre: lustre-OST0000: new disk, initializing [ 184.248694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.259073] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.328582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.316962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.334530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.485274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.705773] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.296154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.419344] Lustre: lustre-OST0001: new disk, initializing [ 209.422964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.436530] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.501212] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.090196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.109804] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 215.238837] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.404026] hrtimer: interrupt took 2548093 ns [ 216.440484] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.252445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.527971] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.491448] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.731693] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing yml_node [ 257.538180] Lustre: DEBUG MARKER: Client: 2.17.57.68 [ 260.060626] Lustre: DEBUG MARKER: MDS: 2.17.57.68 [ 262.769974] Lustre: DEBUG MARKER: OSS: 2.17.57.68 [ 264.415650] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 06:26:56 EDT 2026 [ 282.827260] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 284.624459] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.538893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.556669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.581461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 302.561746] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.565793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 302.585604] Lustre: Skipped 2 previous similar messages [ 302.595684] Lustre: Skipped 3 previous similar messages [ 307.180042] Lustre: server umount lustre-MDT0000 complete [ 311.062123] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826464 with bad export cookie 12820900764267987543 [ 311.064705] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.069094] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 312.804039] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.804094] LustreError: 8454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.804617] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.817218] Lustre: Skipped 1 previous similar message [ 312.868089] LustreError: 8454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 316.833678] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.839597] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.842309] Lustre: Skipped 1 previous similar message [ 316.861331] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 317.715528] Lustre: server umount lustre-MDT0001 complete [ 321.748613] Lustre: server umount lustre-OST0000 complete [ 325.982186] Lustre: server umount lustre-OST0001 complete [ 333.594916] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 344.482380] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 382.638104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.857579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 382.894410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 382.992463] Lustre: lustre-MDT0000: new disk, initializing [ 383.077684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 383.091415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.541604] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 399.439356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.605330] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.703284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.708836] Lustre: Skipped 1 previous similar message [ 399.815384] Lustre: lustre-MDT0001: new disk, initializing [ 400.010665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.073822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 400.100445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 405.497633] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 416.771677] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 423.015927] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 430.642844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 431.181354] Lustre: lustre-OST0000: new disk, initializing [ 431.196872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 431.242409] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 431.409484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.986683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 433.005729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 433.138585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 438.343925] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 448.704148] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 453.239521] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 453.479108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 458.420855] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 458.688515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 463.848960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 463.852584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.858019] LustreError: Skipped 1 previous similar message [ 463.861841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 463.880037] Lustre: Skipped 1 previous similar message [ 463.901802] Lustre: Skipped 2 previous similar messages [ 466.933736] Lustre: server umount lustre-OST0000 complete [ 471.526651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.539275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.085573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.098702] Lustre: Skipped 2 previous similar messages [ 477.470555] Lustre: server umount lustre-MDT0000 complete [ 479.207061] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.229313] LustreError: 15081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 481.387780] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826635 with bad export cookie 12820900764267989076 [ 481.393688] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 481.402326] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 481.714731] Lustre: server umount lustre-MDT0001 complete [ 486.798260] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:30:38 (1787826638) [ 489.681642] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 491.861721] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:30:43 (1787826643) [ 499.833174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.264074] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.334202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.034742] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 505.840608] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.497382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.540621] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 527.779084] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 533.144653] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.935978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.286301] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 552.780184] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 569.825111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 569.829847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.835807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.869946] Lustre: Skipped 2 previous similar messages [ 574.437751] Lustre: server umount lustre-OST0000 complete [ 584.645252] Lustre: server umount lustre-MDT0000 complete [ 585.186380] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.209881] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 588.183051] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826742 with bad export cookie 12820900764267989958 [ 588.184918] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.199440] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 588.519666] Lustre: server umount lustre-MDT0001 complete [ 598.183959] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 600.955685] Key type lgssc unregistered [ 601.308639] LNet: 23011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.341505] LNetError: 23011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.367705] LNet: Removed LNI 192.168.206.133@tcp [ 602.470121] Key type .llcrypt unregistered [ 602.473834] Key type ._llcrypt unregistered [ 624.305760] Key type ._llcrypt registered [ 624.311937] Key type .llcrypt registered [ 624.444385] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 625.979364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 626.001095] alg: No test for adler32 (adler32-zlib) [ 627.068418] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 627.291154] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 628.952281] Key type lgssc registered [ 630.047802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 638.216598] loop: module loaded [ 638.335777] loop0: detected capacity change from 0 to 204800000 [ 638.502589] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.508737] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.775520] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.417529] loop0: detected capacity change from 0 to 204800000 [ 644.623106] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.643947] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.677974] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 649.737953] loop0: detected capacity change from 0 to 204800000 [ 656.086070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 656.119049] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.406278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 657.484731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 657.617247] Lustre: lustre-MDT0000: new disk, initializing [ 657.762133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 657.786095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 662.984952] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 672.029763] loop1: detected capacity change from 0 to 204800000 [ 677.600553] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.936669] Lustre: lustre-OST07c6: new disk, initializing [ 677.940441] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 677.946410] Lustre: Skipped 1 previous similar message [ 677.951670] Lustre: 26201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 678.022871] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 683.774671] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 686.649585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 686.660568] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 686.692702] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 688.960946] Lustre: 25185:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 688.973219] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 688.980890] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 688.987856] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 688.998024] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 689.005099] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 705.373468] Lustre: 25185:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 705.380747] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 705.386889] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 705.394421] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 705.402686] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 705.408906] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 710.159345] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 710.170757] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 710.176407] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 710.195481] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.202736] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 710.209286] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.214267] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 710.220131] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.228488] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 710.241107] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.248971] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 710.259790] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.450792] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 713.457712] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 713.462725] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 713.468049] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.473896] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 713.487551] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.497493] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 713.513820] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.524854] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 713.542455] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.548445] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 713.558078] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.223628] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 718.246948] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 718.259152] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 718.282234] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.298329] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.311455] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.326417] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 718.331167] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.350685] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.356778] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.368815] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.385388] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.735143] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 729.765693] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 729.777068] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 729.784606] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 729.802718] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 729.808783] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 729.823593] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 729.835882] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 729.851661] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 729.860225] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 729.868905] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 729.888434] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 776.444191] Lustre: server umount lustre-OST07c6 complete [ 794.814577] Lustre: server umount lustre-MDT0000 complete [ 811.678815] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 815.271367] Key type lgssc unregistered [ 815.630050] LNet: 28757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.650983] LNetError: 28757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.666601] LNet: Removed LNI 192.168.206.133@tcp [ 816.730676] Key type .llcrypt unregistered [ 816.737571] Key type ._llcrypt unregistered [ 834.834828] Key type ._llcrypt registered [ 834.840581] Key type .llcrypt registered [ 834.943673] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 836.237586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.255053] alg: No test for adler32 (adler32-zlib) [ 837.304871] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 837.539971] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 839.208306] Key type lgssc registered [ 840.343326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 848.240800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 848.273675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.903938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.617794] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 860.132405] LustreError: 30022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.152142] LustreError: 30022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 863.738970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.122702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 868.145984] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 875.035794] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 880.614444] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 886.918213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 887.264758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.022041] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 894.459976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 900.297168] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 918.702783] Lustre: server umount lustre-OST0000 complete [ 925.153723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.169339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 925.175271] Lustre: Skipped 1 previous similar message [ 925.191665] Lustre: Skipped 1 previous similar message [ 930.284166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.297805] Lustre: Skipped 1 previous similar message [ 933.428239] Lustre: server umount lustre-MDT0000 complete [ 934.370036] Lustre: 29334:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827072/real 1787827072] req@00000000c0c1d25f x1874672474945920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787827088 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 934.395555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.414302] LustreError: 31270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.446488] LustreError: 31270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 935.393811] Lustre: 29332:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827073/real 1787827073] req@0000000026ba5b27 x1874672474946688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787827089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 937.305557] LustreError: 33008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827091 with bad export cookie 2211132362466125572 [ 937.308210] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 937.321213] LustreError: 33008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 937.584759] Lustre: server umount lustre-MDT0001 complete [ 946.542546] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 949.048324] Key type lgssc unregistered [ 949.339856] LNet: 33763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.345947] LNetError: 33763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.365154] LNet: Removed LNI 192.168.206.133@tcp [ 950.230529] Key type .llcrypt unregistered [ 950.233984] Key type ._llcrypt unregistered [ 965.793890] Key type ._llcrypt registered [ 965.795946] Key type .llcrypt registered [ 965.896902] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 966.914984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.936930] alg: No test for adler32 (adler32-zlib) [ 967.935582] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 968.168346] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 969.840255] Key type lgssc registered [ 970.962394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.346463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.376835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.813522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.899632] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 989.155840] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.185241] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 991.699075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.136237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.645111] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1005.261109] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1011.010758] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1018.300203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.656525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.840180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1025.233728] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1034.554843] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1050.600110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1050.606789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.634601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.645889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.651945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.688836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.981360] Lustre: server umount lustre-OST0000 complete [ 1062.881387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.882656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.906433] Lustre: Skipped 1 previous similar message [ 1062.916845] Lustre: Skipped 2 previous similar messages [ 1068.002759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.015934] Lustre: Skipped 1 previous similar message [ 1073.123733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1073.134708] Lustre: Skipped 1 previous similar message [ 1074.144243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1074.418313] Lustre: server umount lustre-MDT0000 complete [ 1077.894692] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827231 with bad export cookie 3890875736843664672 [ 1077.915638] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.919714] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1078.335511] Lustre: server umount lustre-MDT0001 complete [ 1088.102237] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 1090.569592] Key type lgssc unregistered [ 1090.900533] LNet: 38574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.908261] LNetError: 38574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.928741] LNet: Removed LNI 192.168.206.133@tcp [ 1091.692845] Key type .llcrypt unregistered [ 1091.700770] Key type ._llcrypt unregistered [ 1114.792914] Key type ._llcrypt registered [ 1114.796150] Key type .llcrypt registered [ 1114.917669] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1116.879945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1116.977380] alg: No test for adler32 (adler32-zlib) [ 1118.254161] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1118.557543] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 1120.229616] Key type lgssc registered [ 1122.189785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1131.181710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1131.200776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.580852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.667970] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1142.760699] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.785769] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1146.120421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1146.552607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1151.194896] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1159.100719] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1165.016786] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1172.587140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1172.917415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.053524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1179.777618] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1188.359268] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1198.053352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1198.058754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.078185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.493340] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.33@tcp (stopping) [ 1201.642378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.645086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.667651] Lustre: Skipped 1 previous similar message [ 1203.818261] Lustre: server umount lustre-MDT0000 complete [ 1205.577598] LustreError: 39982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.754473] LustreError: 39987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.770566] LustreError: 39987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1208.042348] LustreError: 39968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827361 with bad export cookie 12178762621914625209 [ 1208.060112] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.078748] LustreError: 39968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1210.703662] LustreError: 39982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.729512] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.33@tcp (stopping) [ 1210.736863] Lustre: Skipped 2 previous similar messages [ 1211.880657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.638701] Lustre: server umount lustre-MDT0001 complete [ 1230.052353] Lustre: server umount lustre-OST0000 complete [ 1245.755650] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 1248.426413] Key type lgssc unregistered [ 1248.666634] LNet: 43848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.673986] LNetError: 43848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.708678] LNet: Removed LNI 192.168.206.133@tcp [ 1250.364428] Key type .llcrypt unregistered [ 1250.371061] Key type ._llcrypt unregistered [ 1274.410700] Key type ._llcrypt registered [ 1274.415989] Key type .llcrypt registered [ 1274.598937] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1276.686305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1276.756496] alg: No test for adler32 (adler32-zlib) [ 1278.064040] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1278.434311] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 1280.128160] Key type lgssc registered [ 1281.805697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1289.881758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1289.904311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.426630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.192401] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1301.992246] LustreError: 45264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.007843] LustreError: 45264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1304.028239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.291348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1308.408436] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1315.293348] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1319.845543] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1325.691560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.023971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.341197] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1336.309238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1337.815438] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1344.373695] Lustre: lustre-OST0000: Not available for connect from 192.168.206.33@tcp (stopping) [ 1345.508527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1345.523830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.541095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.538599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1346.541126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.580297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.584353] Lustre: Skipped 1 previous similar message [ 1349.473359] LustreError: 47087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.602472] Lustre: server umount lustre-OST0000 complete [ 1371.105749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1371.117211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.139963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.505724] Lustre: server umount lustre-MDT0000 complete [ 1377.381357] LustreError: 45246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827531 with bad export cookie 8895444930890316998 [ 1377.395197] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1383.925187] Lustre: server umount lustre-MDT0001 complete [ 1395.465925] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 1398.229987] Key type lgssc unregistered [ 1398.602209] LNet: 48975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.609104] LNetError: 48975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.633653] LNet: Removed LNI 192.168.206.133@tcp [ 1399.525962] Key type .llcrypt unregistered [ 1399.531588] Key type ._llcrypt unregistered [ 1420.853979] Key type ._llcrypt registered [ 1420.856090] Key type .llcrypt registered [ 1420.978528] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1421.943637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1422.091687] alg: No test for adler32 (adler32-zlib) [ 1423.212925] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1423.482792] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 1425.144159] Key type lgssc registered [ 1426.361896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1433.973940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1434.003583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.584815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.901849] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1445.862696] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.876570] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1449.017513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.474735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.808684] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1464.065508] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1470.916269] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1478.235927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1478.660510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.754697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1485.063503] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1492.420470] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1501.592651] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1504.911718] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1516.985577] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 6h47m48s [ 1518.546899] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1520.609562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.613387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.628173] Lustre: Skipped 2 previous similar messages [ 1520.629439] Lustre: Skipped 1 previous similar message [ 1521.716934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.723722] Lustre: Skipped 3 previous similar messages [ 1525.103873] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.33@tcp (stopping) [ 1526.637331] Lustre: server umount lustre-MDT0000 complete [ 1527.780307] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.798495] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1530.259986] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1532.232625] LustreError: 53057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.059501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.172967] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1537.468720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.544236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1538.515126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1543.612691] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1548.016255] Lustre: DEBUG MARKER: Wait for df (36755) ... [ 1549.773702] Lustre: DEBUG MARKER: done [ 1556.450647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1556.467400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.502758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1556.514140] Lustre: Skipped 1 previous similar message [ 1557.985221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1557.998116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.431497] Lustre: server umount lustre-OST0000 complete [ 1569.256884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.263736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1569.270087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1569.281724] Lustre: Skipped 1 previous similar message [ 1569.287616] Lustre: Skipped 3 previous similar messages [ 1571.078875] Lustre: server umount lustre-MDT0000 complete [ 1574.970818] LustreError: 50352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827728 with bad export cookie 17564481489528702839 [ 1574.972758] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1574.985410] LustreError: 50352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1575.321257] Lustre: server umount lustre-MDT0001 complete [ 1587.221390] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 1590.148990] Key type lgssc unregistered [ 1590.545598] LNet: 55298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.565063] LNetError: 55298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.583159] LNet: Removed LNI 192.168.206.133@tcp [ 1591.769742] Key type .llcrypt unregistered [ 1591.773684] Key type ._llcrypt unregistered [ 1632.106629] Key type ._llcrypt registered [ 1632.109930] Key type .llcrypt registered [ 1632.217621] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1634.143499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1634.169307] alg: No test for adler32 (adler32-zlib) [ 1635.269989] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1635.463088] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 1637.192468] Key type lgssc registered [ 1639.194021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1647.786996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1647.806131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.119650] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1649.150224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1649.342870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.231629] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1663.746102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.817741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1663.840611] Lustre: 57505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1664.031827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1668.274956] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1674.532973] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1679.966424] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1686.357945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.480678] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1686.657406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.740720] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1693.351578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1701.274844] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1709.129343] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 6h51m01s [ 1710.491335] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1718.665505] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1721.313106] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1721.399972] LustreError: 57513:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1741.707214] Lustre: lustre-MDT0000: Client ce5af1fe-a8fb-43a0-b726-74c62c75983e (at 192.168.206.33@tcp) reconnecting [ 1761.488130] LustreError: 57513:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1763.824047] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1766.267557] Lustre: DEBUG MARKER: done [ 1773.526401] Lustre: server umount lustre-OST0000 complete [ 1780.707505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.716989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.725158] Lustre: Skipped 1 previous similar message [ 1781.809919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.816127] Lustre: Skipped 1 previous similar message [ 1787.881215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.891982] Lustre: Skipped 1 previous similar message [ 1787.976042] Lustre: server umount lustre-MDT0000 complete [ 1790.945439] Lustre: 56827:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827928/real 1787827928] req@000000005b488458 x1874673311734528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787827944 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1790.999809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.018428] Lustre: Skipped 1 previous similar message [ 1791.876523] LustreError: 59368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827945 with bad export cookie 15066940374091534731 [ 1791.885574] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1791.889330] LustreError: 59368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1792.290657] Lustre: server umount lustre-MDT0001 complete [ 1803.029649] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 1805.689485] Key type lgssc unregistered [ 1805.988320] LNet: 61838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.994920] LNetError: 61838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.019296] LNet: Removed LNI 192.168.206.133@tcp [ 1806.861809] Key type .llcrypt unregistered [ 1806.863760] Key type ._llcrypt unregistered [ 1850.844979] Key type ._llcrypt registered [ 1850.859746] Key type .llcrypt registered [ 1850.997115] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 1852.050898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.130409] alg: No test for adler32 (adler32-zlib) [ 1853.218387] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1853.470086] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 1855.208886] Key type lgssc registered [ 1856.423744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1859.257229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.272187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.512073] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1860.561231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1860.707313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.751762] Lustre: Failing over lustre-MDT0000 [ 1862.077752] Lustre: server umount lustre-MDT0000 complete [ 1869.143735] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:53:40 (1787828020) [ 1876.116038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.520224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.666493] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1889.123148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.192841] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1889.217434] Lustre: 64632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1889.470049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1893.863409] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1902.147359] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1908.218151] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1914.643399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1914.830754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1915.024644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.443727] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1922.220390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1928.980984] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1935.500323] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1943.023779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.033826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.045492] Lustre: Skipped 1 previous similar message [ 1943.060703] Lustre: Skipped 2 previous similar messages [ 1948.137296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.147376] Lustre: Skipped 2 previous similar messages [ 1949.056853] Lustre: server umount lustre-MDT0000 complete [ 1952.572671] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828106 with bad export cookie 4240953724690478843 [ 1952.578392] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.580231] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1953.072767] Lustre: server umount lustre-MDT0001 complete [ 1956.763853] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1961.185364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1969.632505] Lustre: 63557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828107/real 1787828107] req@00000000e44f24dd x1874673540309504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787828123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.667417] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.680472] Lustre: Skipped 1 previous similar message [ 1969.704654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.729149] Lustre: 63559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828111/real 1787828111] req@00000000319495fe x1874673540309632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787828127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1974.752830] Lustre: 63560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828112/real 1787828112] req@00000000a0a99967 x1874673540309760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787828128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1977.893642] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3adae4f16d0bbfde [ 1977.909078] Lustre: MGC192.168.206.133@tcp: Connection restored to 0@lo (at 0@lo) [ 1978.264661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.375785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1979.241328] Lustre: 63557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828116/real 1787828116] req@000000006a0e92b0 x1874673540310016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787828132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1982.812618] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1983.396900] Lustre: 63557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828121/real 1787828121] req@00000000227b4a1e x1874673540310144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787828137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1984.495192] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.521838] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1989.603514] LustreError: 68404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.634486] LustreError: 68404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1991.607387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.911573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1997.301190] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1997.561894] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 1998.312410] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2005.651565] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2012.658883] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2022.881891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2022.887580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.902636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.613402] Lustre: server umount lustre-MDT0000 complete [ 2029.031393] LustreError: 68403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.051145] LustreError: 68403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2030.180377] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828184 with bad export cookie 4240953724690481118 [ 2030.185446] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2030.199680] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2030.435809] Lustre: server umount lustre-MDT0001 complete [ 2035.962446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2045.076551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.528546] Lustre: 63558:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828188/real 1787828188] req@000000000805e42c x1874673540339200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787828204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2050.548974] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.555949] Lustre: Skipped 2 previous similar messages [ 2055.715105] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3adae4f16d0bc541 [ 2055.724968] Lustre: MGC192.168.206.133@tcp: Connection restored to 0@lo (at 0@lo) [ 2056.060295] LustreError: 66485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2056.172261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.236378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2060.215222] LustreError: 71087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.233450] LustreError: 71087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2060.346661] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2068.760790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.666431] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2074.608499] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2081.329955] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2087.204476] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2095.080180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.089207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2095.110934] Lustre: Skipped 2 previous similar messages [ 2095.127291] Lustre: Skipped 5 previous similar messages [ 2100.195497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.204102] Lustre: Skipped 2 previous similar messages [ 2101.053947] Lustre: server umount lustre-MDT0000 complete [ 2104.756387] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828258 with bad export cookie 4240953724690482497 [ 2104.760579] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.775248] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2105.083404] Lustre: server umount lustre-MDT0001 complete [ 2111.082915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2114.178929] Lustre: DEBUG MARKER: files compared the same [ 2126.041468] Lustre: server umount lustre-OST0000 complete [ 2141.275196] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 2143.812563] Key type lgssc unregistered [ 2144.060834] LNet: 74367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.065837] LNetError: 74367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.079809] LNet: Removed LNI 192.168.206.133@tcp [ 2144.846638] Key type .llcrypt unregistered [ 2144.850914] Key type ._llcrypt unregistered [ 2165.900214] Key type ._llcrypt registered [ 2165.902845] Key type .llcrypt registered [ 2166.013414] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 2166.643743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.661057] alg: No test for adler32 (adler32-zlib) [ 2167.651448] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2167.928261] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 2169.584235] Key type lgssc registered [ 2170.565229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2177.513657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2177.534109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.956390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.394689] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug +malloc all [ 2188.258709] LustreError: 75779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.269667] LustreError: 75779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2192.022397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.382928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.935676] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug +malloc all [ 2204.082940] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2208.893359] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2215.368142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2215.709800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2221.849438] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug +malloc all [ 2223.913350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2228.831979] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2238.432823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2238.440958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.452250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2239.457722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2239.464302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.465277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2239.486924] Lustre: Skipped 1 previous similar message [ 2244.588130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.595697] Lustre: Skipped 1 previous similar message [ 2249.697348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2249.702608] Lustre: Skipped 1 previous similar message [ 2251.233033] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2251.387800] Lustre: server umount lustre-OST0000 complete [ 2254.817416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.829780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.839375] Lustre: Skipped 1 previous similar message [ 2260.985335] Lustre: server umount lustre-MDT0000 complete [ 2264.447803] LustreError: 77622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828418 with bad export cookie 12221495557104437767 [ 2264.449339] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2264.465302] LustreError: 77622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2264.752117] Lustre: server umount lustre-MDT0001 complete [ 2273.480223] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 2275.835617] Key type lgssc unregistered [ 2276.063508] LNet: 79313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.077284] LNetError: 79313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.097245] LNet: Removed LNI 192.168.206.133@tcp [ 2276.820443] Key type .llcrypt unregistered [ 2276.826164] Key type ._llcrypt unregistered [ 2297.681230] Key type ._llcrypt registered [ 2297.684558] Key type .llcrypt registered [ 2297.829427] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 2299.263597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.322552] alg: No test for adler32 (adler32-zlib) [ 2300.548168] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2300.917456] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 2302.616183] Key type lgssc registered [ 2303.785959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2313.134956] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2313.168843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2328.929073] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2334.048324] LustreError: 80697:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.133@tcp: failed processing log, type 4: rc = -110 [ 2360.736346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.626565] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2374.264876] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2381.297550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.392427] Lustre: *** cfs_fail_loc=706, val=0*** [ 2381.759887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.842335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2385.565580] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2386.917825] LustreError: 80723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.933835] LustreError: 80723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2392.039118] LustreError: 80722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.678914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.192030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2398.231487] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2405.497272] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2411.275749] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2414.566933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2414.576738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.603124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2416.103108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2416.108973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.942737] Lustre: server umount lustre-OST0000 complete [ 2424.801366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2424.820111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.842075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.849219] Lustre: Skipped 1 previous similar message [ 2431.458360] LustreError: 81788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.471369] LustreError: 81788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2432.168228] Lustre: server umount lustre-MDT0000 complete [ 2435.887435] LustreError: 80704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828589 with bad export cookie 7033927088344613088 [ 2435.910480] LustreError: 80704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2435.913200] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.175736] Lustre: server umount lustre-MDT0001 complete [ 2448.123267] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 2450.979777] Key type lgssc unregistered [ 2451.320831] LNet: 84315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.335583] LNetError: 84315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.411383] LNet: Removed LNI 192.168.206.133@tcp [ 2453.219757] Key type .llcrypt unregistered [ 2453.224968] Key type ._llcrypt unregistered [ 2478.605729] Key type ._llcrypt registered [ 2478.611247] Key type .llcrypt registered [ 2478.698896] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 2479.510355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.582552] alg: No test for adler32 (adler32-zlib) [ 2480.597719] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2480.792910] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 2482.432173] Key type lgssc registered [ 2483.616625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2491.219881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2491.256713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.572733] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2505.118360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2505.617787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2510.023385] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2510.818991] LustreError: 86423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.835100] LustreError: 86423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2515.940724] LustreError: 86427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.956683] LustreError: 86427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2518.405630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.875777] LustreError: 86423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.911843] LustreError: 86423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2518.931170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.131265] LustreError: 87168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.149577] LustreError: 87168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2525.031364] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2529.252045] LustreError: 86421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.282333] LustreError: 86421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2533.118814] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2533.128949] Lustre: MGS: Client 2e678f77-64a4-4010-b40c-4b96c351a9dd (at 0@lo) reconnecting [ 2533.154047] LustreError: 86437:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2533.397876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.472283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2537.954659] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2553.829917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2553.835863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.844752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2553.867710] Lustre: Skipped 1 previous similar message [ 2553.889587] Lustre: Skipped 1 previous similar message [ 2557.657919] Lustre: server umount lustre-OST0000 complete [ 2564.065284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.066611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2564.100916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2564.122041] LustreError: Skipped 1 previous similar message [ 2569.186945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.202822] Lustre: Skipped 2 previous similar messages [ 2573.282877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.298665] Lustre: Skipped 1 previous similar message [ 2576.352184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2576.533290] Lustre: server umount lustre-MDT0000 complete [ 2578.403902] LustreError: 86426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.427752] LustreError: 86426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2580.981829] Lustre: server umount lustre-MDT0001 complete [ 2585.522349] Lustre: server umount MGS complete [ 2597.194638] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:05:49 (1787828749) [ 2627.786526] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 2637.043911] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 2667.204970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.157577] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2680.656880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.986742] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2681.062861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2681.164722] Lustre: lustre-MDT0001: new disk, initializing [ 2681.229983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.259215] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2681.264876] Lustre: 93593:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2681.271175] LustreError: 93593:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2686.878905] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2698.397330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.672244] Lustre: lustre-OST0000: new disk, initializing [ 2698.678145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2698.686422] Lustre: 94359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2698.756038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.665642] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2705.959581] Lustre: 94733:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004fd4f7af x1874674197766528/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787828875 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2713.745862] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2720.439852] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2720.454727] Lustre: MGS: Client e214d539-bda7-4c41-aecf-08fca6ee92d6 (at 0@lo) reconnecting [ 2720.498460] Lustre: 95404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2720.516353] LustreError: 93579:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2720.543087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2720.645174] Lustre: lustre-MDT0000: new disk, initializing [ 2720.762425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.787707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2720.814757] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2721.839626] Lustre: 95428:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f32cba12 x1874674197773056/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787828891 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2725.533160] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2727.072931] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2731.001616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2731.016964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2731.151621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2757.091034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2757.101961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.123273] Lustre: Skipped 1 previous similar message [ 2757.129898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.147236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.152342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.891396] Lustre: server umount lustre-OST0000 complete [ 2769.377828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2769.383754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.395607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2769.395615] Lustre: Skipped 2 previous similar messages [ 2769.425844] Lustre: Skipped 1 previous similar message [ 2771.717238] Lustre: server umount lustre-MDT0000 complete [ 2774.502437] LustreError: 93565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.525335] LustreError: 93565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2781.920386] Lustre: server umount lustre-MDT0001 complete [ 2785.884696] Lustre: server umount MGS complete [ 2797.851964] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:09:10 (1787828950) [ 2815.961620] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 2818.480609] Key type lgssc unregistered [ 2818.759446] LNet: 97929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2818.768668] LNetError: 97929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2818.793680] LNet: Removed LNI 192.168.206.133@tcp [ 2819.627742] Key type .llcrypt unregistered [ 2819.629841] Key type ._llcrypt unregistered [ 2835.451339] Key type ._llcrypt registered [ 2835.453317] Key type .llcrypt registered [ 2835.532313] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 2836.612090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2836.698058] alg: No test for adler32 (adler32-zlib) [ 2837.711050] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2837.959665] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 2839.632159] Key type lgssc registered [ 2840.759099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2848.397408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2848.419429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.016173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.410172] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2859.496884] LustreError: 99144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.532244] LustreError: 99144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2863.842766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.353475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2869.355029] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2877.092210] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.471291] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2891.374738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.984265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.053288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2899.582647] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 2907.774511] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2919.920559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.925169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2919.948661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2919.962939] Lustre: Skipped 1 previous similar message [ 2922.885521] Lustre: server umount lustre-OST0000 complete [ 2930.150115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.176153] Lustre: Skipped 1 previous similar message [ 2930.194454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.211199] Lustre: Skipped 1 previous similar message [ 2933.000760] Lustre: server umount lustre-MDT0000 complete [ 2935.266327] LustreError: 99144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.291397] LustreError: 99144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2936.632869] LustreError: 99123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829090 with bad export cookie 14706471547688633065 [ 2936.639908] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.651533] LustreError: 99123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2937.146759] Lustre: server umount lustre-MDT0001 complete [ 2949.416177] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 2952.553338] Key type lgssc unregistered [ 2952.887946] LNet: 102680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2952.891729] LNetError: 102680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.900256] LNet: Removed LNI 192.168.206.133@tcp [ 2953.896291] Key type .llcrypt unregistered [ 2953.898324] Key type ._llcrypt unregistered [ 2973.829951] Key type ._llcrypt registered [ 2973.838503] Key type .llcrypt registered [ 2973.956696] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 2975.019488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2975.055844] alg: No test for adler32 (adler32-zlib) [ 2976.455111] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2976.730221] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 2978.376296] Key type lgssc registered [ 2979.469927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2986.493413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2986.499758] LustreError: 103759:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2986.556361] LustreError: 103766:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2986.576094] LustreError: 103759:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4939 [ 2986.594804] LustreError: 103759:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2986.599173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.605506] LustreError: 103759:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2988.349745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.256653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.741519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3002.698507] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3008.454821] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing lsmod [ 3012.171658] LustreError: 104970:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3012.193077] LustreError: 104973:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3012.204785] LustreError: 104970:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4973 [ 3012.213488] LustreError: 104970:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3012.227993] LustreError: 104970:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3012.231351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3012.707898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.760053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3018.210365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3018.223349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.255609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.233600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.239540] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.233713] Lustre: server umount lustre-MDT0000 complete [ 3022.377962] LustreError: 103804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.412261] LustreError: 103804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3026.325028] LustreError: 105011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829180 with bad export cookie 7852100833577528925 [ 3026.340478] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.342333] LustreError: 105011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3026.552258] LustreError: 104234:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000909dab70 x1874674717839872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3026.742192] Lustre: server umount lustre-MDT0001 complete [ 3040.625856] Lustre: server umount lustre-OST0000 complete [ 3051.139936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.689552] LustreError: 106037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.769989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3057.133136] LustreError: 106038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.356810] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3061.227953] LustreError: 106037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.345282] LustreError: 106038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.199870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.666396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3073.498644] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3082.226372] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3088.110279] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3094.342505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3094.586994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.635731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3100.829811] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3108.677623] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3122.146495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3122.159530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.177412] Lustre: Skipped 1 previous similar message [ 3122.188159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.198949] Lustre: Skipped 2 previous similar messages [ 3127.373963] Lustre: server umount lustre-OST0000 complete [ 3133.413427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.420259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.423341] Lustre: Skipped 2 previous similar messages [ 3133.438300] Lustre: Skipped 3 previous similar messages [ 3137.473612] Lustre: server umount lustre-MDT0000 complete [ 3138.541386] LustreError: 106773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.569228] LustreError: 106773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3141.105510] LustreError: 106019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829294 with bad export cookie 7852100833577529835 [ 3141.119813] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3141.121593] LustreError: 106019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3141.501376] Lustre: server umount lustre-MDT0001 complete [ 3153.839691] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3156.784503] Key type lgssc unregistered [ 3157.078114] LNet: 109602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3157.085563] LNetError: 109602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3157.122589] LNet: Removed LNI 192.168.206.133@tcp [ 3158.129603] Key type .llcrypt unregistered [ 3158.131809] Key type ._llcrypt unregistered [ 3183.855652] Key type ._llcrypt registered [ 3183.868703] Key type .llcrypt registered [ 3183.997606] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3184.797071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3184.935072] alg: No test for adler32 (adler32-zlib) [ 3186.031726] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3186.232613] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 3187.896257] Key type lgssc registered [ 3189.045611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3197.549982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3197.568594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.140685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.938513] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3208.681204] LustreError: 111031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.700547] LustreError: 111031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3212.635117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.078755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3217.772684] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3224.890849] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3230.427871] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3237.192299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.674447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.592901] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3243.831926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3251.401427] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3256.931128] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3261.372090] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3269.609678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3269.610265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.611311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3269.644986] Lustre: Skipped 1 previous similar message [ 3272.449120] Lustre: server umount lustre-OST0000 complete [ 3279.843292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3279.854609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.903903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.911934] Lustre: Skipped 2 previous similar messages [ 3282.765675] Lustre: server umount lustre-MDT0000 complete [ 3284.966959] LustreError: 112039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.004112] LustreError: 112039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3287.301757] LustreError: 111012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829441 with bad export cookie 4592902566714139957 [ 3287.329549] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.354056] LustreError: 111012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3287.879792] Lustre: server umount lustre-MDT0001 complete [ 3301.019238] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3303.611868] Key type lgssc unregistered [ 3303.894751] LNet: 114674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3303.907209] LNetError: 114674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3303.925627] LNet: Removed LNI 192.168.206.133@tcp [ 3304.665801] Key type .llcrypt unregistered [ 3304.668061] Key type ._llcrypt unregistered [ 3323.261506] Key type ._llcrypt registered [ 3323.264313] Key type .llcrypt registered [ 3323.394265] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3324.686811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.721141] alg: No test for adler32 (adler32-zlib) [ 3326.026685] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3326.245152] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 3327.912170] Key type lgssc registered [ 3329.464832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3338.150164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3338.189933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.753653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.166759] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3349.991612] LustreError: 115892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3350.024755] LustreError: 115892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3354.629193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.234788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3360.996841] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3369.297513] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3376.124510] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3383.793082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.316515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.326405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3390.488410] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3398.370808] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3409.891836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3409.902485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.919594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3413.454329] Lustre: server umount lustre-OST0000 complete [ 3420.138593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.159590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3420.160487] Lustre: Skipped 2 previous similar messages [ 3420.188937] Lustre: Skipped 2 previous similar messages [ 3423.654261] Lustre: server umount lustre-MDT0000 complete [ 3425.257492] LustreError: 117006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.278884] LustreError: 117006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3427.718811] LustreError: 115874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829581 with bad export cookie 9938943309100717848 [ 3427.729079] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3427.730849] LustreError: 115874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3428.084242] Lustre: server umount lustre-MDT0001 complete [ 3440.240391] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3443.186663] Key type lgssc unregistered [ 3443.561694] LNet: 119430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3443.570347] LNetError: 119430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3443.587783] LNet: Removed LNI 192.168.206.133@tcp [ 3444.548070] Key type .llcrypt unregistered [ 3444.553106] Key type ._llcrypt unregistered [ 3473.205610] Key type ._llcrypt registered [ 3473.208058] Key type .llcrypt registered [ 3473.354947] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3475.058066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3475.337734] alg: No test for adler32 (adler32-zlib) [ 3476.846530] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3477.209786] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 3478.985032] Key type lgssc registered [ 3481.208462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3490.252721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3490.296526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.879311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3496.868754] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3502.052374] LustreError: 120852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.081858] LustreError: 120852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3506.670715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.116073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3511.773497] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3519.779594] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3526.660529] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3534.202504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3534.574191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.628372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3540.974996] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3548.676601] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3559.479224] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3564.122575] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3564.134426] Lustre: Skipped 2 previous similar messages [ 3567.741140] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3567.814096] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3569.777087] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3569.785061] Lustre: Skipped 2 previous similar messages [ 3575.839971] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3575.844500] Lustre: Skipped 1 previous similar message [ 3581.572823] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.33@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3581.595836] Lustre: Skipped 2 previous similar messages [ 3586.130958] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.33@tcp 192.168.0.[2,10]@tcp [ 3592.161823] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.33/24@tcp in log lustre-MDT0000 [ 3592.168636] Lustre: Skipped 2 previous similar messages [ 3593.783917] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.33/24@tcp [ 3593.788463] Lustre: Skipped 1 previous similar message [ 3608.375903] Lustre: server umount lustre-OST0000 complete [ 3613.666224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.684314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3613.690962] Lustre: Skipped 1 previous similar message [ 3618.794405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.807796] Lustre: Skipped 1 previous similar message [ 3623.095117] Lustre: server umount lustre-MDT0000 complete [ 3623.907747] LustreError: 120851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.927335] LustreError: 120851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3624.928257] Lustre: 120162:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829762/real 1787829762] req@000000000bfd6585 x1874675243223424/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787829778 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3624.949209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.968486] Lustre: Skipped 1 previous similar message [ 3624.983229] LustreError: 120852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.008786] LustreError: 120852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3627.399875] LustreError: 120831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829781 with bad export cookie 18037415793258985072 [ 3627.414373] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.425179] LustreError: 120831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3627.842706] Lustre: server umount lustre-MDT0001 complete [ 3640.682689] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3644.204477] Key type lgssc unregistered [ 3644.572931] LNet: 125705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3644.593648] LNetError: 125705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3644.625698] LNet: Removed LNI 192.168.206.133@tcp [ 3645.912273] Key type .llcrypt unregistered [ 3645.915238] Key type ._llcrypt unregistered [ 3675.494332] Key type ._llcrypt registered [ 3675.500093] Key type .llcrypt registered [ 3675.664062] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3677.325958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3677.478101] alg: No test for adler32 (adler32-zlib) [ 3678.612481] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3678.832776] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 3680.568407] Key type lgssc registered [ 3682.229718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3692.415612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3692.436653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.962933] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3693.994140] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.33/24@tcp [ 3694.020786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.313596] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3703.273515] LustreError: 127266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3703.298724] LustreError: 127266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3707.205577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.676786] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3707.704950] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.33/24@tcp [ 3707.811430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3712.609884] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3721.130394] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3727.461310] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3734.036062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3734.472841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.549859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3740.848719] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3748.253636] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3762.153969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3762.162675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.176423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.171264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.182153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.183296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.116035] Lustre: server umount lustre-OST0000 complete [ 3773.410924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.413723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.433664] Lustre: Skipped 1 previous similar message [ 3773.447303] Lustre: Skipped 2 previous similar messages [ 3777.735276] Lustre: server umount lustre-MDT0000 complete [ 3778.534882] LustreError: 128472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.554900] LustreError: 128472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3781.549978] LustreError: 127247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829935 with bad export cookie 10856833244844170756 [ 3781.553210] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3781.559474] LustreError: 127247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3781.943976] Lustre: server umount lustre-MDT0001 complete [ 3791.821461] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 3793.788521] Key type lgssc unregistered [ 3794.107714] LNet: 130860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3794.115806] LNetError: 130860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3794.128690] LNet: Removed LNI 192.168.206.133@tcp [ 3794.754908] Key type .llcrypt unregistered [ 3794.759055] Key type ._llcrypt unregistered [ 3821.051564] Key type ._llcrypt registered [ 3821.055474] Key type .llcrypt registered [ 3821.148588] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 3835.789358] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 3836.374704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3836.455291] alg: No test for adler32 (adler32-zlib) [ 3837.519222] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 3837.738901] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 3839.394969] Key type lgssc registered [ 3840.424358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3876.031878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3876.069992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.344556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3877.404069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3877.541680] Lustre: lustre-MDT0000: new disk, initializing [ 3877.623360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.654272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3881.937718] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3892.444557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.534640] Lustre: 134264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3892.613789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3892.619184] Lustre: Skipped 1 previous similar message [ 3892.701291] Lustre: lustre-MDT0001: new disk, initializing [ 3892.814507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3892.873933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3892.878163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3896.948873] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3906.213668] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3911.977891] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3919.315028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.509921] Lustre: lustre-OST0000: new disk, initializing [ 3919.513203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3919.518682] Lustre: 136233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3919.599837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.252919] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3926.592314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3926.609045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3926.703837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3934.728522] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3939.602784] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3939.816113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3944.641466] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3944.889101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3947.498714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3947.500702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3947.531378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3947.541824] LustreError: Skipped 1 previous similar message [ 3947.566684] Lustre: Skipped 1 previous similar message [ 3952.613870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3952.620955] Lustre: Skipped 1 previous similar message [ 3957.739096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.749642] Lustre: Skipped 1 previous similar message [ 3961.313409] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3961.650191] Lustre: server umount lustre-OST0000 complete [ 3967.975590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.999166] Lustre: Skipped 1 previous similar message [ 3968.009938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3968.023368] Lustre: Skipped 1 previous similar message [ 3971.916323] Lustre: server umount lustre-MDT0000 complete [ 3975.662256] LustreError: 134256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830129 with bad export cookie 18094098228039055107 [ 3975.674709] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3975.676348] LustreError: 134256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3976.228892] Lustre: server umount lustre-MDT0001 complete [ 3984.693956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.561019] LustreError: 138450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.576985] LustreError: 138450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3985.672711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.025604] LustreError: 138449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.070607] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 3995.175695] LustreError: 138450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4000.234233] LustreError: 138449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4001.037529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.657869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4007.502876] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4015.743129] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4021.771898] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4028.718318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4029.098917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.250246] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4046.100445] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4052.957353] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4053.334240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4059.422252] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4059.753152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4069.605199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4069.823182] Lustre: lustre-OST0001: new disk, initializing [ 4069.825732] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4069.833632] Lustre: 141821:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4069.928687] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4076.643520] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4076.697541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4076.710609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4076.835112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4086.625814] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4086.875545] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4091.627689] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4091.804825] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4097.539573] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b318447e800.ost_server_uuid 50 [ 4099.276839] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b318447e800.ost_server_uuid in IDLE state after 0 sec [ 4107.752712] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4107.755930] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4107.765112] LustreError: Skipped 1 previous similar message [ 4107.778053] Lustre: Skipped 1 previous similar message [ 4107.787131] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4107.805824] Lustre: Skipped 1 previous similar message [ 4109.815887] Lustre: server umount lustre-OST0001 complete [ 4112.873929] LustreError: 140270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.913702] LustreError: 140270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4117.990208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4117.994914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4118.010910] Lustre: Skipped 2 previous similar messages [ 4118.019928] Lustre: Skipped 1 previous similar message [ 4123.109494] LustreError: 140272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.125936] LustreError: 140272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4128.229808] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4128.462624] Lustre: server umount lustre-OST0000 complete [ 4133.349087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.360516] Lustre: Skipped 1 previous similar message [ 4138.476888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4138.495511] Lustre: Skipped 7 previous similar messages [ 4138.653159] Lustre: server umount lustre-MDT0000 complete [ 4142.999422] LustreError: 143124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830296 with bad export cookie 18094098228039055989 [ 4143.008694] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4143.016181] LustreError: 143124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4143.463479] Lustre: server umount lustre-MDT0001 complete [ 4163.988099] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 4166.942230] Key type lgssc unregistered [ 4167.329439] LNet: 144839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4167.342586] LNetError: 144839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4167.361952] LNet: Removed LNI 192.168.206.133@tcp [ 4168.456521] Key type .llcrypt unregistered [ 4168.459491] Key type ._llcrypt unregistered [ 4209.817569] Key type ._llcrypt registered [ 4209.820691] Key type .llcrypt registered [ 4209.941824] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 4225.105866] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4226.586598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4226.630342] alg: No test for adler32 (adler32-zlib) [ 4227.684847] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 4227.949925] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 4229.608287] Key type lgssc registered [ 4231.092098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4272.019133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4272.054482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.316364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4273.399539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4273.514414] Lustre: lustre-MDT0000: new disk, initializing [ 4273.611794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.702978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4279.082325] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4290.093604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.354560] Lustre: 148996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.511647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4290.522601] Lustre: Skipped 1 previous similar message [ 4290.631254] Lustre: lustre-MDT0001: new disk, initializing [ 4290.780129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4290.834663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4290.854735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4296.315808] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4309.706726] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4318.062797] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4325.275504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4325.515836] Lustre: lustre-OST0000: new disk, initializing [ 4325.519341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4325.523768] Lustre: 150969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4325.625479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.692787] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4332.667281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4332.684980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4332.803145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4340.715758] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4345.818650] Lustre: Failing over lustre-OST0000 [ 4345.998298] Lustre: server umount lustre-OST0000 complete [ 4363.745145] Lustre: 147221:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830501/real 1787830501] req@000000001d407d4f x1874676029621632/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787830517 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4363.769530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.768103] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830502/real 1787830502] req@000000009c8b71fe x1874676029622656/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787830518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4364.773905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.804593] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4365.339161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.651207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.679671] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4369.697975] Lustre: 147219:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830507/real 1787830507] req@00000000dbb756cb x1874676029623808/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787830523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4369.754368] Lustre: 147219:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4371.256622] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4373.802555] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4374.816313] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830512/real 1787830512] req@0000000039fe38d8 x1874676029625088/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787830528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4374.845564] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4374.923858] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4374.925314] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4375.380666] Lustre: Failing over lustre-MDT0000 [ 4375.691953] Lustre: server umount lustre-MDT0000 complete [ 4378.102277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4378.123311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.147228] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.186293] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4378.933025] LustreError: 149003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.968213] LustreError: 149003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4379.936652] Lustre: 147220:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830517/real 1787830517] req@00000000547b67f1 x1874676029626112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787830533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4384.072577] LustreError: 149003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.100146] LustreError: 149003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4389.198517] LustreError: 150367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.223539] LustreError: 150367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4394.320488] LustreError: 149003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.345492] LustreError: 149003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4394.467328] Lustre: 147221:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830532/real 1787830532] req@0000000055d519a0 x1874676029637760/t0(0) o400->MGC192.168.206.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787830548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4394.497294] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.104449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.708878] LustreError: 150366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.714957] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7eba1d17e6f36924 [ 4404.744238] LustreError: 150366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4404.782423] Lustre: MGC192.168.206.133@tcp: Connection restored to 0@lo (at 0@lo) [ 4404.796955] Lustre: Skipped 1 previous similar message [ 4405.197821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.271682] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4405.581679] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4409.681143] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4410.364274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4410.387755] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4410.432656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4420.583707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.588216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4420.599270] Lustre: Skipped 3 previous similar messages [ 4420.616533] Lustre: Skipped 1 previous similar message [ 4423.806367] Lustre: server umount lustre-OST0000 complete [ 4430.818385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.824576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4430.845091] Lustre: Skipped 1 previous similar message [ 4430.870384] Lustre: Skipped 1 previous similar message [ 4433.539989] Lustre: server umount lustre-MDT0000 complete [ 4435.938703] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4435.968035] LustreError: 149008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4437.100303] LustreError: 148989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830590 with bad export cookie 9131643182943267108 [ 4437.107062] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.119743] LustreError: 148989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4437.499541] Lustre: server umount lustre-MDT0001 complete [ 4447.343756] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 4449.788742] Key type lgssc unregistered [ 4450.210720] LNet: 154627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4450.223190] LNetError: 154627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4450.265184] LNet: Removed LNI 192.168.206.133@tcp [ 4451.360854] Key type .llcrypt unregistered [ 4451.363489] Key type ._llcrypt unregistered [ 4487.749836] Key type ._llcrypt registered [ 4487.751573] Key type .llcrypt registered [ 4487.890175] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4488.657250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4488.755847] alg: No test for adler32 (adler32-zlib) [ 4489.844500] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 4490.155578] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 4491.872150] Key type lgssc registered [ 4493.056886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4501.407944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4501.421818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.787777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.173401] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4512.239874] LustreError: 156146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.266482] LustreError: 156146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4517.349961] LustreError: 156147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.723855] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4521.833643] LustreError: 156175:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4521.857502] LustreError: 156175:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4521.862897] LustreError: 156175:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4522.427867] Lustre: server umount lustre-MDT0000 complete [ 4538.037630] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 4547.340734] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 4588.633737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.929438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4588.971756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4589.052566] Lustre: lustre-MDT0000: new disk, initializing [ 4589.116989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.129703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4595.253511] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4607.268811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.468626] Lustre: 159481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4607.580412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4607.585373] Lustre: Skipped 1 previous similar message [ 4607.702381] Lustre: lustre-MDT0001: new disk, initializing [ 4607.836535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4607.906541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4607.928575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4613.033208] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4622.787601] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4627.532463] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4633.636022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4634.000510] Lustre: lustre-OST0000: new disk, initializing [ 4634.007173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4634.019417] Lustre: 161447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4634.145472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.322518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4640.340601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4640.510133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4640.796590] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 4650.268374] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4654.636919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4656.633351] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:40:08 (1787830808) [ 4912.097510] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.101068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4912.116884] Lustre: Skipped 1 previous similar message [ 4912.144410] Lustre: Skipped 2 previous similar messages [ 4916.673680] Lustre: server umount lustre-MDT0000 complete [ 4917.217702] LustreError: 159489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4917.252835] LustreError: 159489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4920.661112] LustreError: 159474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831074 with bad export cookie 15390586623303431314 [ 4920.664756] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4920.676176] LustreError: 159474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4922.337637] LustreError: 159488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.337749] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.376318] Lustre: Skipped 1 previous similar message [ 4922.379485] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4925.344978] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4927.029558] Lustre: server umount lustre-MDT0001 complete [ 4945.376270] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4945.580993] Lustre: server umount lustre-OST0000 complete [ 4956.702742] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 4967.710236] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 5002.751114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.961327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5002.993111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5003.135297] Lustre: lustre-MDT0000: new disk, initializing [ 5003.243266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5003.256754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5007.639811] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5018.521983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5018.677990] Lustre: 166112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5018.728104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5018.738490] Lustre: Skipped 1 previous similar message [ 5018.907809] Lustre: lustre-MDT0001: new disk, initializing [ 5018.992320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5019.034385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5019.056025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5023.522940] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5032.762666] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5038.784306] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5044.429039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5044.701996] Lustre: lustre-OST0000: new disk, initializing [ 5044.706019] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5044.712747] Lustre: 168079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5044.783175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.857474] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5051.460625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5051.466738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5051.533987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5060.003377] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5064.189928] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5064.349683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5068.495976] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5068.707266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5071.852333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.882789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5071.885774] Lustre: Skipped 1 previous similar message [ 5076.439720] Lustre: server umount lustre-OST0000 complete [ 5080.551076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5080.564502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5080.580173] Lustre: Skipped 1 previous similar message [ 5080.588763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5085.591175] Lustre: server umount lustre-MDT0000 complete [ 5087.202501] LustreError: 166123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.231696] LustreError: 166123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5089.033211] LustreError: 168078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831242 with bad export cookie 15390586623303590144 [ 5089.038773] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5089.048046] LustreError: 168078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5089.349965] Lustre: server umount lustre-MDT0001 complete [ 5096.664460] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:47:29 (1787831249) [ 5112.849980] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 5121.806695] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 5153.888595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.091636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5154.115766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5154.174150] Lustre: lustre-MDT0000: new disk, initializing [ 5154.233410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5154.257655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5158.226635] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5168.444213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.559638] Lustre: 172832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5168.748150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5173.149086] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5182.102687] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5187.888791] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5196.313460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5196.834960] Lustre: lustre-OST0000: new disk, initializing [ 5196.849636] Lustre: Skipped 1 previous similar message [ 5196.857620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5196.875569] Lustre: Skipped 2 previous similar messages [ 5196.909884] Lustre: 174811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5203.052965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5203.063966] Lustre: Skipped 1 previous similar message [ 5203.068929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5203.196674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5204.417609] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5213.231561] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5219.715763] Lustre: server umount lustre-OST0000 complete [ 5223.909662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.912699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5223.918018] Lustre: Skipped 2 previous similar messages [ 5223.923759] Lustre: Skipped 2 previous similar messages [ 5233.241455] Lustre: server umount lustre-MDT0000 complete [ 5234.149970] LustreError: 172846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.178289] LustreError: 172846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5237.292514] LustreError: 172825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831391 with bad export cookie 15390586623303591026 [ 5237.301838] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5237.317529] LustreError: 172825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5248.101671] Lustre: server umount lustre-MDT0001 complete [ 5255.718605] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:50:07 (1787831407) [ 5271.103400] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 5278.937836] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 5307.636332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.864047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5307.896908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5307.971352] Lustre: lustre-MDT0000: new disk, initializing [ 5308.033811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.037837] Lustre: Skipped 2 previous similar messages [ 5308.050861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5312.085841] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5321.382914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.507682] Lustre: 179137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5321.832516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5325.066486] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5334.561171] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5340.428621] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5346.696151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5346.962927] Lustre: 181113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5353.159139] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5356.641177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5356.769494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5362.478590] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5372.387089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.400204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5372.404649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5372.416150] Lustre: Skipped 4 previous similar messages [ 5376.067852] Lustre: server umount lustre-OST0000 complete [ 5385.864808] Lustre: server umount lustre-MDT0000 complete [ 5387.745375] LustreError: 179148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.762889] LustreError: 179148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5389.056570] LustreError: 179128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831542 with bad export cookie 15390586623303592104 [ 5389.066180] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5389.073850] LustreError: 179128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5389.422474] Lustre: server umount lustre-MDT0001 complete [ 5399.352812] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 5401.591736] Key type lgssc unregistered [ 5401.810681] LNet: 182974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5401.817365] LNetError: 182974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5401.828587] LNet: Removed LNI 192.168.206.133@tcp [ 5402.515515] Key type .llcrypt unregistered [ 5402.517552] Key type ._llcrypt unregistered [ 5425.980666] Key type ._llcrypt registered [ 5425.983714] Key type .llcrypt registered [ 5426.107995] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 5427.170694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5427.409203] alg: No test for adler32 (adler32-zlib) [ 5428.522399] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 5428.758740] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 5430.441354] Key type lgssc registered [ 5431.560596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5439.023307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5439.040194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.388550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.610057] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5450.726761] LustreError: 184390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5450.756472] LustreError: 184390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5454.141071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5454.535784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5459.006394] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5466.905532] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5472.539913] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5479.986800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.414206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5485.579947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5487.009310] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5493.278428] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5515.242447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5515.249885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.261932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5520.354339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5520.359523] Lustre: Skipped 1 previous similar message [ 5520.732044] Lustre: server umount lustre-OST0000 complete [ 5525.489355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.504203] Lustre: Skipped 1 previous similar message [ 5525.512617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5525.520226] Lustre: Skipped 2 previous similar messages [ 5530.379659] Lustre: server umount lustre-MDT0000 complete [ 5530.600636] LustreError: 184390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5530.616332] LustreError: 184390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5533.771369] LustreError: 184371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831687 with bad export cookie 17411819314745978042 [ 5533.782876] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5533.783805] LustreError: 184371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5534.056448] Lustre: server umount lustre-MDT0001 complete [ 5543.945228] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 5546.462698] Key type lgssc unregistered [ 5546.713779] LNet: 187946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5546.722079] LNetError: 187946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5546.746528] LNet: Removed LNI 192.168.206.133@tcp [ 5547.420529] Key type .llcrypt unregistered [ 5547.428791] Key type ._llcrypt unregistered [ 5567.177739] Key type ._llcrypt registered [ 5567.181551] Key type .llcrypt registered [ 5567.278679] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 5568.515963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5568.539133] alg: No test for adler32 (adler32-zlib) [ 5569.599947] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 5569.847375] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 5571.504154] Key type lgssc registered [ 5572.496162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5580.118523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5580.143371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.617908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.399189] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5592.054603] LustreError: 189344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5592.064091] LustreError: 189344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5593.231193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.514306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5597.729884] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5605.336439] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5611.576944] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5617.791497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5618.106846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.584912] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5626.313786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5630.877524] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5639.019032] Lustre: lustre-OST0000: Not available for connect from 192.168.206.33@tcp (stopping) [ 5639.659243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5639.680245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.707976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5641.705656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5641.710198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5641.710726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5641.729480] Lustre: Skipped 1 previous similar message [ 5644.174662] LustreError: 191168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5644.386628] Lustre: server umount lustre-OST0000 complete [ 5652.441158] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b31856c3000.ost_server_uuid 50 [ 5654.196672] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5670.373364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5670.388607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.404185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5673.982053] Lustre: server umount lustre-MDT0000 complete [ 5676.521333] LustreError: 190073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5676.544073] LustreError: 190073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5677.459935] LustreError: 189326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831831 with bad export cookie 7693751025334103411 [ 5677.466972] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5677.472781] LustreError: 189326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5677.909188] Lustre: server umount lustre-MDT0001 complete [ 5687.654476] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:57:19 (1787831839) [ 5695.338895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5695.652808] LustreError: 193262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.770380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5700.456595] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5701.090318] LustreError: 193263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5710.273246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5710.306937] LustreError: 193262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5710.321431] LustreError: 193262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5710.680764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5715.149911] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5723.322755] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5728.554324] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5735.419885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5735.703316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5740.847177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5742.453907] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5750.405885] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5758.134340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5758.434108] Lustre: lustre-OST0001: new disk, initializing [ 5758.437978] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5758.453798] Lustre: 196104:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5758.603174] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5765.366407] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5765.897713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5765.927681] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5766.214123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5775.860329] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5780.962918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5780.974268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.982362] Lustre: Skipped 1 previous similar message [ 5780.985428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5780.988880] Lustre: Skipped 2 previous similar messages [ 5785.263745] Lustre: server umount lustre-OST0000 complete [ 5786.606133] LustreError: 195079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5786.625280] LustreError: 195079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5791.147592] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5791.312970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5794.839711] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5794.993955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5812.193538] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5812.203174] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.207528] LustreError: Skipped 1 previous similar message [ 5812.245850] Lustre: Skipped 2 previous similar messages [ 5812.246659] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5812.265854] Lustre: Skipped 4 previous similar messages [ 5813.533777] Lustre: server umount lustre-OST0001 complete [ 5831.137697] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5831.423232] Lustre: server umount lustre-MDT0000 complete [ 5832.674841] LustreError: 193263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.686422] LustreError: 193263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 5834.675882] LustreError: 193243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831988 with bad export cookie 7693751025334104482 [ 5834.678870] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5834.684483] LustreError: 193243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5835.175248] Lustre: server umount lustre-MDT0001 complete [ 5857.393459] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:00:09 (1787832009) [ 5863.512780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5863.723050] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5863.750445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5863.947025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.818662] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5875.890962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.981067] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5876.027053] Lustre: 199538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5880.576372] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5887.303555] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5892.437301] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5898.570581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5898.717785] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5904.185209] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5907.051929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5910.676615] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5916.630575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5916.696112] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5922.699961] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 5927.036793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5929.036540] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5932.520787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.522651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5932.523942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5932.523950] Lustre: Skipped 6 previous similar messages [ 5932.534710] Lustre: Skipped 3 previous similar messages [ 5937.269145] Lustre: server umount lustre-OST0000 complete [ 5937.646875] LustreError: 203017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.662888] LustreError: 203017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5952.999846] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.608910] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5958.743896] Lustre: server umount lustre-OST0001 complete [ 5968.356724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5968.361911] Lustre: Skipped 9 previous similar messages [ 5968.732594] Lustre: server umount lustre-MDT0000 complete [ 5971.965695] LustreError: 199531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832125 with bad export cookie 7693751025334105889 [ 5971.981799] LustreError: 199531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5971.981939] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5972.256569] Lustre: server umount lustre-MDT0001 complete [ 5993.630216] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:02:25 (1787832145) [ 6010.222612] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 6019.779694] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 6053.305521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6053.562323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6053.581960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6053.661008] Lustre: lustre-MDT0000: new disk, initializing [ 6053.756666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6053.768632] Lustre: Skipped 3 previous similar messages [ 6053.810799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6058.364705] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6068.430916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.541238] Lustre: 207657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6068.612984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6068.619330] Lustre: Skipped 1 previous similar message [ 6068.711387] Lustre: lustre-MDT0001: new disk, initializing [ 6068.879885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6068.887719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6072.870646] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6081.953583] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6086.515588] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6092.868777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.090330] Lustre: lustre-OST0000: new disk, initializing [ 6093.093570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6093.098120] Lustre: 209621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6098.973946] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6098.990289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6098.997559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6099.072878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6108.632079] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6114.356234] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6114.579491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6120.185605] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6120.415944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6124.515494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6124.524425] LustreError: Skipped 1 previous similar message [ 6124.525251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6124.531462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6124.554202] Lustre: Skipped 4 previous similar messages [ 6124.565101] Lustre: Skipped 1 previous similar message [ 6128.873617] Lustre: server umount lustre-OST0000 complete [ 6139.873349] LustreError: 207668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6139.886138] LustreError: 207668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6140.822194] Lustre: server umount lustre-MDT0000 complete [ 6144.357147] LustreError: 207647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832298 with bad export cookie 7693751025334107674 [ 6144.360502] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6144.367936] LustreError: 207647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6144.664259] Lustre: server umount lustre-MDT0001 complete [ 6154.166592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.235652] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6168.249724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.075651] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6180.454507] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6186.937644] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6195.354643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6195.863629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6195.877055] Lustre: Skipped 4 previous similar messages [ 6203.317929] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6211.015978] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6216.213682] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6216.442415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6222.022281] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6222.321096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6225.891264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6225.908522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6225.918723] Lustre: Skipped 2 previous similar messages [ 6231.281838] Lustre: server umount lustre-OST0000 complete [ 6239.134619] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6244.398686] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6275.106329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.637518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.651092] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6282.511715] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6291.817034] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6293.337137] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6294.056603] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.082930] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6304.235860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6304.248929] Lustre: Skipped 6 previous similar messages [ 6306.661498] Lustre: server umount lustre-OST0000 complete [ 6318.023644] Lustre: server umount lustre-MDT0000 complete [ 6322.590034] LustreError: 211819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832476 with bad export cookie 7693751025334108556 [ 6322.599771] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6322.615824] LustreError: 211819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6323.151837] Lustre: server umount lustre-MDT0001 complete [ 6332.161446] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:08:04 (1787832484) [ 6339.275772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.826814] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6352.365932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.617569] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6364.600408] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6370.191608] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6377.903713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.814060] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6394.555566] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6399.575960] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6399.896370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6405.292022] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6405.653618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6415.160095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6415.317286] Lustre: lustre-OST0001: new disk, initializing [ 6415.321290] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6415.327677] Lustre: 221177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6420.653297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6420.673546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6420.786934] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6423.104729] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6432.767987] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6437.630397] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6437.826134] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6442.014282] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6442.337903] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6446.561228] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6446.570229] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.571823] LustreError: Skipped 4 previous similar messages [ 6446.596317] Lustre: Skipped 6 previous similar messages [ 6450.540122] Lustre: server umount lustre-OST0001 complete [ 6451.686519] LustreError: 219632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.707971] LustreError: 219632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6456.995704] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6457.195876] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6461.715487] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6461.874983] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6492.111509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.307171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.321107] Lustre: Skipped 5 previous similar messages [ 6497.780752] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6497.812625] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6497.817116] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6498.737915] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6506.333361] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6507.923056] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6524.384328] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6524.471204] Lustre: server umount lustre-OST0001 complete [ 6547.607371] Lustre: server umount lustre-OST0000 complete [ 6558.019697] Lustre: server umount lustre-MDT0000 complete [ 6563.144542] LustreError: 217790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832717 with bad export cookie 7693751025334109522 [ 6563.146740] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6563.172155] LustreError: 217790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6569.744336] Lustre: server umount lustre-MDT0001 complete [ 6593.329343] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:12:25 (1787832745) [ 6598.998124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6599.229586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6599.257836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6602.815467] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6610.715146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.828036] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6610.870988] Lustre: 226426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6614.542569] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6620.879868] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6626.359102] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6633.022737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.201702] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6639.774496] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6648.211287] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6657.395316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6657.451049] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6663.773963] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6672.418886] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6674.521922] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6679.618622] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6679.823181] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6685.253084] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6685.409668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6691.698674] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b31856c4800.ost_server_uuid 50 [ 6693.062994] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b31856c4800.ost_server_uuid in IDLE state after 0 sec [ 6694.024860] Lustre: Permanently deactivating lustre-OST0001 [ 6694.040768] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6697.555795] Lustre: Permanently reactivating lustre-OST0001 [ 6703.073059] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6703.074566] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6703.083781] LustreError: Skipped 4 previous similar messages [ 6703.085312] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6703.094140] Lustre: Skipped 7 previous similar messages [ 6703.097124] Lustre: Skipped 16 previous similar messages [ 6707.351709] Lustre: server umount lustre-OST0001 complete [ 6717.325711] Lustre: server umount lustre-OST0000 complete [ 6735.328128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6735.560474] Lustre: server umount lustre-MDT0000 complete [ 6739.333151] LustreError: 229295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832893 with bad export cookie 7693751025334110810 [ 6739.340957] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.348541] LustreError: 229295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6739.628297] Lustre: server umount lustre-MDT0001 complete [ 6763.338345] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:15:14 (1787832914) [ 6770.737740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.040089] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6771.089198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6771.097339] Lustre: Skipped 5 previous similar messages [ 6775.474157] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6782.752661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.840916] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6782.870571] Lustre: 233116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6787.574301] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6794.753431] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6800.550768] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6807.587727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.842394] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6807.881401] Lustre: Permanently deactivating lustre-OST0000 [ 6807.901000] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6813.961234] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6817.859996] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6820.790198] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6826.587696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6826.641735] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6827.349419] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6832.810941] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6839.599113] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6841.309271] Lustre: Permanently reactivating lustre-OST0000 [ 6841.319841] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6841.323966] Lustre: Skipped 2 previous similar messages [ 6847.616341] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6847.627060] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6847.642583] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6847.653897] Lustre: Skipped 1 previous similar message [ 6847.716726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6847.743502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6860.492699] Lustre: server umount lustre-OST0001 complete [ 6870.722868] Lustre: server umount lustre-OST0000 complete [ 6880.906430] Lustre: server umount lustre-MDT0000 complete [ 6884.753377] LustreError: 233107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833038 with bad export cookie 7693751025334112455 [ 6884.757133] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6884.760364] LustreError: 233107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6885.149671] Lustre: server umount lustre-MDT0001 complete [ 6895.734854] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 6898.566729] Key type lgssc unregistered [ 6898.855753] LNet: 238000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6898.877456] LNetError: 238000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6899.952618] LNet: Removed LNI 192.168.206.133@tcp [ 6900.866440] Key type .llcrypt unregistered [ 6900.868135] Key type ._llcrypt unregistered [ 6922.462430] Key type ._llcrypt registered [ 6922.464490] Key type .llcrypt registered [ 6922.597584] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 6923.972373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6923.991531] alg: No test for adler32 (adler32-zlib) [ 6925.019724] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 6925.190940] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 6926.840812] Key type lgssc registered [ 6928.004824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6938.312441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6938.346393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.923728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.083776] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6949.412271] LustreError: 239469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6949.440900] LustreError: 239469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6954.477964] LustreError: 239468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6954.632412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.851914] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6954.894352] Lustre: 239457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6954.903744] Lustre: Permanently deactivating lustre-MDT0001 [ 6954.927813] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6955.357332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6960.647856] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6961.681573] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6969.426687] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6975.263982] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6981.951640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6982.198263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.244188] LustreError: 241309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.261685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6988.530877] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 6989.291043] LustreError: 241309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.417483] LustreError: 241442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.340420] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7003.151590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7003.327538] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7008.787990] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7008.791345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7009.353389] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7017.007033] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7020.933809] Lustre: Permanently reactivating lustre-MDT0001 [ 7020.942480] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7020.946294] Lustre: Skipped 2 previous similar messages [ 7023.152049] Lustre: lustre-MDT0001: Client 7050696f-36d9-4671-9444-9f6c4d9b24d0 (at 192.168.206.33@tcp) reconnecting [ 7028.447799] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7029.865108] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.876037] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7029.887377] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7029.897037] LustreError: 243247:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7029.900083] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7029.905130] LustreError: 243247:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7029.920361] LustreError: 243247:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.133@tcp: cfg command failed: rc = -108 [ 7029.929555] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7029.929555] [ 7029.940101] LustreError: 239461:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7030.062471] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7034.885514] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7035.092593] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7039.033470] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7039.212949] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7041.483368] Lustre: Permanently deactivating lustre-MDT0001 [ 7041.494754] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7041.497483] Lustre: Skipped 2 previous similar messages [ 7047.737955] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7060.961858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.969949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7060.973585] Lustre: Skipped 3 previous similar messages [ 7060.983065] Lustre: Skipped 3 previous similar messages [ 7064.510834] Lustre: server umount lustre-MDT0000 complete [ 7066.085340] LustreError: 239463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7066.102520] LustreError: 239463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7067.848894] LustreError: 241329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833221 with bad export cookie 6568959147869141761 [ 7067.855083] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.857478] LustreError: 241329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7068.158261] Lustre: server umount lustre-MDT0001 complete [ 7077.652984] Lustre: server umount lustre-OST0000 complete [ 7088.839402] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:20:40 (1787833240) [ 7098.082392] Lustre: server umount lustre-OST0001 complete [ 7104.084077] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 7112.082382] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7146.748695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7147.044787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7147.062529] Lustre: Skipped 2 previous similar messages [ 7147.084940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7147.195952] Lustre: lustre-MDT0000: new disk, initializing [ 7147.309892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.349620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7151.903463] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7163.421208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.527718] Lustre: 247747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7163.585975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7163.589640] Lustre: Skipped 1 previous similar message [ 7163.706828] Lustre: lustre-MDT0001: new disk, initializing [ 7163.819243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.876674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7163.909410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7169.325994] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7178.861527] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7184.682959] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7190.676604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7190.952186] Lustre: lustre-OST0000: new disk, initializing [ 7190.955846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7190.964399] Lustre: 249714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7191.050950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.766442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7196.789434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7196.882269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7197.155210] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7205.774219] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7207.894817] LustreError: 247756:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7216.413933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.654807] Lustre: lustre-OST0001: new disk, initializing [ 7216.661516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7216.676875] Lustre: 250833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7217.984234] LustreError: 247756:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7218.018879] LustreError: 250605:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7222.336238] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7224.469675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7224.492346] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7224.520258] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7228.096176] LustreError: 250605:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7230.832966] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7232.351519] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7235.061221] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7235.065841] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7235.068900] LustreError: Skipped 1 previous similar message [ 7235.074120] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7235.085862] Lustre: Skipped 1 previous similar message [ 7235.110794] Lustre: Skipped 1 previous similar message [ 7240.167347] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7240.192049] Lustre: Skipped 2 previous similar messages [ 7240.522533] LustreError: 249699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.592957] Lustre: server umount lustre-OST0001 complete [ 7258.599441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7258.607351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7258.612184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7258.621375] Lustre: Skipped 1 previous similar message [ 7258.636685] Lustre: Skipped 1 previous similar message [ 7262.966577] Lustre: server umount lustre-OST0000 complete [ 7266.274048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7266.281029] LustreError: Skipped 1 previous similar message [ 7266.289678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.302573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7266.308626] Lustre: Skipped 2 previous similar messages [ 7272.487205] Lustre: server umount lustre-MDT0000 complete [ 7276.005171] LustreError: 247759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.023493] LustreError: 247759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7276.147700] LustreError: 250831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833430 with bad export cookie 6568959147869144358 [ 7276.150020] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7276.167464] LustreError: 250831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7282.558663] Lustre: server umount lustre-MDT0001 complete [ 7293.188801] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 7295.694697] Key type lgssc unregistered [ 7296.011996] LNet: 252867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7296.026651] LNetError: 252867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7296.054084] LNet: Removed LNI 192.168.206.133@tcp [ 7296.874673] Key type .llcrypt unregistered [ 7296.877287] Key type ._llcrypt unregistered [ 7336.846845] Key type ._llcrypt registered [ 7336.848797] Key type .llcrypt registered [ 7336.933443] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7337.777666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7337.820530] alg: No test for adler32 (adler32-zlib) [ 7338.812200] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7339.011070] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 7340.665630] Key type lgssc registered [ 7341.586722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7348.274629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7348.290657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7349.624602] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7349.649365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7349.919260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.815579] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7361.912566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7362.001808] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7362.034477] Lustre: 255081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7362.386684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7367.127448] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7373.177823] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7378.056221] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7384.031748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.249589] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7384.452857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.105216] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7392.890782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7396.755686] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7449.058210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7449.068481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.080458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7449.571038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7449.581843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.603219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7453.060630] Lustre: server umount lustre-OST0000 complete [ 7457.951326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7471.487898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7471.924063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7476.926234] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7483.784953] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7489.261177] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7489.276636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7489.277285] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7489.309707] Lustre: Skipped 1 previous similar message [ 7525.345622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7525.356744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7525.369916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7525.374619] Lustre: Skipped 1 previous similar message [ 7525.394796] Lustre: Skipped 1 previous similar message [ 7529.737933] Lustre: server umount lustre-OST0000 complete [ 7535.589918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.593377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7535.607520] Lustre: Skipped 1 previous similar message [ 7535.620567] Lustre: Skipped 1 previous similar message [ 7540.706085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7540.715756] Lustre: Skipped 1 previous similar message [ 7548.384099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7548.534296] Lustre: server umount lustre-MDT0000 complete [ 7552.302576] LustreError: 255072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833706 with bad export cookie 4478075407018721739 [ 7552.311476] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7552.315592] LustreError: 255072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7552.828898] Lustre: server umount lustre-MDT0001 complete [ 7563.705300] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 7566.361867] Key type lgssc unregistered [ 7566.692349] LNet: 260507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7566.698634] LNetError: 260507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7566.727977] LNet: Removed LNI 192.168.206.133@tcp [ 7567.687271] Key type .llcrypt unregistered [ 7567.690117] Key type ._llcrypt unregistered [ 7592.169654] Key type ._llcrypt registered [ 7592.177287] Key type .llcrypt registered [ 7592.297769] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7593.456371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7593.488894] alg: No test for adler32 (adler32-zlib) [ 7594.619930] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7594.882087] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 7596.560363] Key type lgssc registered [ 7597.630401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7605.488527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7605.528432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.027986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.913827] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7616.482628] LustreError: 261924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7616.497687] LustreError: 261924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7619.363483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7619.645747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.609699] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7632.420967] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7638.446272] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7644.704300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7645.164120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7651.317860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7651.997728] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7659.591321] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7682.019804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7682.023594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.045114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7682.060358] Lustre: Skipped 1 previous similar message [ 7682.079250] Lustre: Skipped 1 previous similar message [ 7687.142433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7687.154405] Lustre: Skipped 1 previous similar message [ 7688.519926] Lustre: server umount lustre-OST0000 complete [ 7696.363315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7696.376218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.390607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7697.380406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.349627] Lustre: server umount lustre-MDT0000 complete [ 7702.505164] LustreError: 261919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7702.520780] LustreError: 261919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7703.489892] LustreError: 261904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833857 with bad export cookie 17829316410856271246 [ 7703.496748] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7703.517101] LustreError: 261904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7703.964331] Lustre: server umount lustre-MDT0001 complete [ 7714.121315] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 7716.859529] Key type lgssc unregistered [ 7717.091675] LNet: 266230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7717.098672] LNetError: 266230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7717.124535] LNet: Removed LNI 192.168.206.133@tcp [ 7718.123164] Key type .llcrypt unregistered [ 7718.124834] Key type ._llcrypt unregistered [ 7735.286554] Key type ._llcrypt registered [ 7735.293316] Key type .llcrypt registered [ 7735.403906] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7736.395738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7736.406914] alg: No test for adler32 (adler32-zlib) [ 7737.517223] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7737.767951] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 7739.448239] Key type lgssc registered [ 7740.571121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7748.084679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7748.126934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.568430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.626139] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7759.844166] LustreError: 267451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7759.868384] LustreError: 267451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7762.033245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.363138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7766.084382] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7773.400622] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7778.863245] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7785.475991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7785.942669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7792.180519] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7795.117833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7799.507963] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7813.607051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.612525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.630853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7815.653489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7815.657554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.658259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7815.666860] Lustre: Skipped 1 previous similar message [ 7818.688275] Lustre: server umount lustre-OST0000 complete [ 7823.849631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7823.862571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.879996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.896451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.903264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.921437] Lustre: Skipped 1 previous similar message [ 7829.811870] Lustre: server umount lustre-MDT0000 complete [ 7831.009531] LustreError: 267451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.047071] LustreError: 267451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7833.670065] LustreError: 270378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833987 with bad export cookie 2742933339076135598 [ 7833.683386] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7833.690321] LustreError: 270378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7833.990887] Lustre: server umount lustre-MDT0001 complete [ 7845.548546] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 7848.418317] Key type lgssc unregistered [ 7848.769659] LNet: 271132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7848.781084] LNetError: 271132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7849.830933] LNet: Removed LNI 192.168.206.133@tcp [ 7850.664676] Key type .llcrypt unregistered [ 7850.667049] Key type ._llcrypt unregistered [ 7875.267259] Key type ._llcrypt registered [ 7875.268948] Key type .llcrypt registered [ 7875.385197] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 7876.742038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7876.771439] alg: No test for adler32 (adler32-zlib) [ 7878.018704] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7878.292763] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 7879.976157] Key type lgssc registered [ 7881.080102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7889.308310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7889.339747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7890.620052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7895.817658] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7901.155532] LustreError: 272559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.183405] LustreError: 272559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7905.188594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7905.636032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7910.311860] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7918.370460] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7924.399539] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7931.295025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.565353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.691211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7937.655576] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 7944.686327] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7972.834770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7972.849423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.861185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7976.937087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7976.943763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7976.950418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7976.968212] Lustre: Skipped 1 previous similar message [ 7978.265670] Lustre: server umount lustre-OST0000 complete [ 7982.052091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.065760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7982.071200] Lustre: Skipped 1 previous similar message [ 7987.172257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7987.177232] Lustre: Skipped 1 previous similar message [ 7987.860270] Lustre: server umount lustre-MDT0000 complete [ 7991.768071] LustreError: 272538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834145 with bad export cookie 15481950808574681921 [ 7991.774491] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7991.781532] LustreError: 272538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7992.096228] Lustre: server umount lustre-MDT0001 complete [ 8002.142411] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 8004.612228] Key type lgssc unregistered [ 8004.843583] LNet: 276869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8004.856924] LNetError: 276869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8004.887341] LNet: Removed LNI 192.168.206.133@tcp [ 8005.599506] Key type .llcrypt unregistered [ 8005.601437] Key type ._llcrypt unregistered [ 8022.880381] Key type ._llcrypt registered [ 8022.882668] Key type .llcrypt registered [ 8022.986616] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8023.778755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8023.827764] alg: No test for adler32 (adler32-zlib) [ 8024.849475] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 8025.117986] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 8026.800166] Key type lgssc registered [ 8027.764698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8034.795548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8034.829971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.273501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.752280] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8046.570059] LustreError: 278105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8046.612227] LustreError: 278105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8050.190866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.568578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8054.614199] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8062.498285] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8068.453853] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8075.098773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.174125] LustreError: 279948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8075.362626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.265163] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8081.390315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8088.156526] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8100.836434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8100.859148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.873515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8101.330975] Lustre: server umount lustre-OST0000 complete [ 8106.979809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.983118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8107.002939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8107.016957] Lustre: Skipped 1 previous similar message [ 8112.099049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8112.108320] Lustre: Skipped 1 previous similar message [ 8116.196058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.196058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8119.264167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8119.470625] Lustre: server umount lustre-MDT0000 complete [ 8121.315771] LustreError: 278108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.337594] LustreError: 278108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8123.536354] LustreError: 278070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834277 with bad export cookie 12975155834794841083 [ 8123.537774] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8123.555543] LustreError: 278070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8123.940459] Lustre: server umount lustre-MDT0001 complete [ 8135.247372] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 8137.942772] Key type lgssc unregistered [ 8138.263566] LNet: 281810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8138.274902] LNetError: 281810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8138.294453] LNet: Removed LNI 192.168.206.133@tcp [ 8139.100770] Key type .llcrypt unregistered [ 8139.105460] Key type ._llcrypt unregistered [ 8151.798189] Key type ._llcrypt registered [ 8151.802577] Key type .llcrypt registered [ 8151.898149] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8169.769864] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 8186.818833] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8187.969402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8188.114743] alg: No test for adler32 (adler32-zlib) [ 8189.282138] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 8189.544692] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 8191.249459] Key type lgssc registered [ 8192.588261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8224.543382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8224.565396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.707224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8225.730743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8225.784824] Lustre: lustre-MDT0000: new disk, initializing [ 8225.857929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.878233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8229.375386] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8238.604660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.722142] Lustre: 285363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8238.766762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8238.771762] Lustre: Skipped 1 previous similar message [ 8238.847408] Lustre: lustre-MDT0001: new disk, initializing [ 8238.915766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8238.955270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8238.973785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8242.970963] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8252.324862] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8257.992375] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8264.172810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8264.406873] Lustre: lustre-OST0000: new disk, initializing [ 8264.411615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8264.422758] Lustre: 287331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8264.502412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.043387] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8271.943877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8271.958455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8272.085457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8280.926558] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8285.928910] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8286.162941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8291.468066] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8291.655885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8297.954487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8297.954653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.977183] LustreError: Skipped 1 previous similar message [ 8297.977992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8298.016804] Lustre: Skipped 1 previous similar message [ 8300.337933] Lustre: server umount lustre-OST0000 complete [ 8305.634271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8305.651055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8305.699077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8305.709895] Lustre: Skipped 1 previous similar message [ 8311.137884] Lustre: server umount lustre-MDT0000 complete [ 8315.346600] LustreError: 285358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834469 with bad export cookie 9712030864124143036 [ 8315.359449] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8315.360388] LustreError: 285358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8315.564968] Lustre: server umount lustre-MDT0001 complete [ 8324.709118] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:41:16 (1787834476) [ 8330.569673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.099758] LustreError: 289745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8331.131029] LustreError: 289745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8331.187538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.411320] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8336.355529] LustreError: 289746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.450760] LustreError: 289745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8342.742955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8343.115894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8346.958905] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8353.875272] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8359.613315] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8366.861408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8367.278732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8373.000946] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8380.091422] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8389.090617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8389.101073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.111528] Lustre: Skipped 1 previous similar message [ 8389.120810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8389.133335] Lustre: Skipped 2 previous similar messages [ 8391.139216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8391.142982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8391.161278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.002781] Lustre: server umount lustre-OST0000 complete [ 8399.329872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8399.337434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8399.348856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8399.358128] Lustre: Skipped 1 previous similar message [ 8403.659275] Lustre: server umount lustre-MDT0000 complete [ 8406.498138] LustreError: 289745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.518824] LustreError: 289745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8406.903441] LustreError: 292241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834560 with bad export cookie 9712030864124143918 [ 8406.910152] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8406.913151] LustreError: 292241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8407.183914] Lustre: server umount lustre-MDT0001 complete [ 8416.483068] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 8419.027303] Key type lgssc unregistered [ 8419.390614] LNet: 293284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8419.397983] LNetError: 293284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8419.408825] LNet: Removed LNI 192.168.206.133@tcp [ 8420.174129] Key type .llcrypt unregistered [ 8420.180033] Key type ._llcrypt unregistered [ 8446.797312] Key type ._llcrypt registered [ 8446.800673] Key type .llcrypt registered [ 8446.891383] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 8461.807134] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8462.837520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8462.904752] alg: No test for adler32 (adler32-zlib) [ 8463.991918] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 8464.219042] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 8465.888185] Key type lgssc registered [ 8467.021654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8513.874322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8513.926828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8515.133581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8515.167476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8515.259793] Lustre: lustre-MDT0000: new disk, initializing [ 8515.349544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.362503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8520.110365] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8530.186622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.302612] Lustre: 297110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8530.364050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8530.371114] Lustre: Skipped 1 previous similar message [ 8530.456815] Lustre: lustre-MDT0001: new disk, initializing [ 8530.550786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8530.590807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8530.602792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8534.835490] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8543.995578] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8549.596786] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8555.423854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8555.641197] Lustre: lustre-OST2710: new disk, initializing [ 8555.644660] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8555.651156] Lustre: 299076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8555.728291] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8561.091918] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8561.228958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8561.247035] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8561.345034] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8569.941935] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8576.701302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8576.902502] Lustre: lustre-OST03e8: new disk, initializing [ 8576.906219] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8576.928339] Lustre: 300141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8577.072165] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8583.339216] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8583.808060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8583.823549] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8583.977095] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8591.061736] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8598.955474] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8599.166847] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8604.143694] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8604.352793] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8609.549922] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8609.767101] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8615.365745] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8615.697178] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8620.001415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8620.011150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.015515] Lustre: Skipped 3 previous similar messages [ 8625.122163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8625.135710] Lustre: Skipped 3 previous similar messages [ 8625.953127] Lustre: server umount lustre-MDT0000 complete [ 8629.589828] LustreError: 297105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834783 with bad export cookie 1323575527027282826 [ 8629.602091] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8629.921561] Lustre: server umount lustre-MDT0001 complete [ 8644.486238] Lustre: server umount lustre-OST2710 complete [ 8646.626416] Lustre: 294909:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834784/real 1787834784] req@00000000ed5a3b3e x1874680472471552/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787834800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8646.672607] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8646.693723] Lustre: Skipped 3 previous similar messages [ 8648.926902] Lustre: server umount lustre-OST03e8 complete [ 8669.626900] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 8679.101665] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8712.338533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8712.556961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8712.581217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8712.667312] Lustre: lustre-MDT0000: new disk, initializing [ 8712.726183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8712.739569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8716.623065] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8726.506359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8726.727515] Lustre: 305392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8726.798437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8726.804682] Lustre: Skipped 1 previous similar message [ 8726.880131] Lustre: lustre-MDT0001: new disk, initializing [ 8726.963469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8727.015728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8727.033729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8730.998773] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8739.188877] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8744.212918] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8749.822781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8750.069662] Lustre: lustre-OST0000: new disk, initializing [ 8750.074650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8750.082662] Lustre: 307363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8750.171056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8754.916416] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8756.804406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8756.817096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8756.890049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8763.155491] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8766.840809] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8767.037174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8770.850402] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8771.055842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8777.697807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8777.702814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8777.716795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8777.725906] Lustre: Skipped 1 previous similar message [ 8778.902601] Lustre: server umount lustre-OST0000 complete [ 8782.820986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8782.831260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8782.832410] Lustre: Skipped 2 previous similar messages [ 8782.839842] Lustre: Skipped 1 previous similar message [ 8787.932241] Lustre: server umount lustre-MDT0000 complete [ 8790.748455] LustreError: 305383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834944 with bad export cookie 1323575527027284506 [ 8790.751504] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8790.755204] LustreError: 305383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8791.017314] Lustre: server umount lustre-MDT0001 complete [ 8797.874878] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:49:10 (1787834950) [ 8799.181463] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8800.622563] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:49:12 (1787834952) [ 8803.522967] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8826.237604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8826.508878] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8826.538411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8826.656287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.395559] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8839.291669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.427320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8839.475641] Lustre: 311218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8844.024623] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8866.363846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8866.596450] Lustre: Denying initial registration attempt from nid 192.168.206.133@tcp, specified as failover [ 8866.603119] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8866.609864] LustreError: 312669:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8866.617747] LustreError: 312669:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8866.623061] LustreError: 312669:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8866.670782] Lustre: server umount lustre-OST0000 complete [ 8866.672981] LustreError: 312669:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8874.981472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8874.994974] Lustre: Skipped 1 previous similar message [ 8874.996130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8875.011018] Lustre: Skipped 1 previous similar message [ 8878.749228] Lustre: server umount lustre-MDT0000 complete [ 8880.102726] LustreError: 311230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8880.124301] LustreError: 311230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8882.441258] LustreError: 311211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835036 with bad export cookie 1323575527027285388 [ 8882.445603] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8882.454891] LustreError: 311211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8882.852348] Lustre: server umount lustre-MDT0001 complete [ 8893.548557] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 8896.139575] Key type lgssc unregistered [ 8896.457563] LNet: 313794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8896.468960] LNetError: 313794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8896.480453] LNet: Removed LNI 192.168.206.133@tcp [ 8897.298551] Key type .llcrypt unregistered [ 8897.303322] Key type ._llcrypt unregistered [ 8908.163920] Key type ._llcrypt registered [ 8908.169300] Key type .llcrypt registered [ 8908.265469] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8909.427775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8909.457167] alg: No test for adler32 (adler32-zlib) [ 8910.539308] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 8910.730765] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 8912.376234] Key type lgssc registered [ 8913.438950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8959.734386] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 8969.319336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8969.341610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.707878] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8970.753090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8970.979548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8975.545048] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 8984.305371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8984.531915] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8984.604985] Lustre: 316571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8984.883602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8990.043324] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9002.148807] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9011.367318] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9032.496462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9032.677294] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9032.823251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9038.982748] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9046.550079] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9050.593825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9050.606146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.622206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9050.627325] Lustre: Skipped 1 previous similar message [ 9054.855821] Lustre: server umount lustre-OST0000 complete [ 9060.834580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9060.850286] Lustre: Skipped 1 previous similar message [ 9060.858921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9060.866241] Lustre: Skipped 1 previous similar message [ 9064.520219] Lustre: server umount lustre-MDT0000 complete [ 9068.382296] LustreError: 318440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835222 with bad export cookie 12041769830282769098 [ 9068.396379] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9068.403994] LustreError: 318440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9068.835788] Lustre: server umount lustre-MDT0001 complete [ 9079.085201] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 9081.469163] Key type lgssc unregistered [ 9081.729843] LNet: 320150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9081.736171] LNetError: 320150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9081.770517] LNet: Removed LNI 192.168.206.133@tcp [ 9082.554077] Key type .llcrypt unregistered [ 9082.556295] Key type ._llcrypt unregistered [ 9104.554287] Key type ._llcrypt registered [ 9104.557911] Key type .llcrypt registered [ 9104.661341] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 9105.689362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9105.712097] alg: No test for adler32 (adler32-zlib) [ 9106.742480] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 9106.969370] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 9108.624501] Key type lgssc registered [ 9109.544978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9116.338244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9116.351936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9117.779847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9121.527266] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9127.908289] LustreError: 321568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.937193] LustreError: 321568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9128.478842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9128.753632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9132.831260] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9139.669468] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9145.532024] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9154.542994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9154.973958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9160.257410] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9166.566872] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9185.254814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9185.265138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9185.278264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9188.714178] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.33@tcp (stopping) [ 9188.840304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.847287] Lustre: Skipped 1 previous similar message [ 9190.511760] Lustre: server umount lustre-MDT0000 complete [ 9193.803670] LustreError: 321563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.189270] LustreError: 321549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835348 with bad export cookie 13135893990895748024 [ 9194.222056] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9198.925293] LustreError: 323111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9198.963933] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.33@tcp (stopping) [ 9198.967923] LustreError: 323111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9198.988921] Lustre: Skipped 3 previous similar messages [ 9199.078896] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.845895] Lustre: server umount lustre-MDT0001 complete [ 9209.263456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9220.823359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9241.423854] LustreError: 323401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9241.445211] LustreError: 323401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9241.511723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9241.576746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9245.539393] LustreError: 323402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9246.300976] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9254.889676] LustreError: 323401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9254.913227] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9254.923164] LustreError: 323401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9255.661646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.249883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9262.145519] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9264.636523] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9271.591506] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9273.381526] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9279.793501] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9281.541476] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9287.137653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9287.142187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9287.168110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9287.173548] Lustre: Skipped 1 previous similar message [ 9292.004544] Lustre: server umount lustre-OST0000 complete [ 9295.329386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.337713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9295.339077] Lustre: Skipped 2 previous similar messages [ 9295.348272] Lustre: Skipped 4 previous similar messages [ 9301.580325] Lustre: server umount lustre-MDT0000 complete [ 9305.344530] LustreError: 321550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835459 with bad export cookie 13135893990895755871 [ 9305.354035] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9305.358068] LustreError: 321550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9305.859336] Lustre: server umount lustre-MDT0001 complete [ 9315.402492] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 9317.885613] Key type lgssc unregistered [ 9318.198883] LNet: 327864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9318.203533] LNetError: 327864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9318.230970] LNet: Removed LNI 192.168.206.133@tcp [ 9319.029511] Key type .llcrypt unregistered [ 9319.032327] Key type ._llcrypt unregistered [ 9340.832878] Key type ._llcrypt registered [ 9340.836413] Key type .llcrypt registered [ 9340.937854] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 9341.669855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9341.730539] alg: No test for adler32 (adler32-zlib) [ 9342.828281] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 9343.048541] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 9344.704172] Key type lgssc registered [ 9345.645098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9352.517683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9352.532617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.921047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.354253] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9363.431482] LustreError: 329277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.442647] LustreError: 329277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9364.571271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9364.873166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9368.603559] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9374.876982] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9380.269076] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9383.398466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9383.406646] Lustre: Skipped 1 previous similar message [ 9383.409548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9383.418503] Lustre: Skipped 1 previous similar message [ 9388.513559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9388.518035] Lustre: Skipped 1 previous similar message [ 9389.539557] LustreError: 330006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9389.663771] Lustre: server umount lustre-MDT0000 complete [ 9392.669746] LustreError: 329258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835546 with bad export cookie 4489619061713340875 [ 9392.684870] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9392.898945] Lustre: server umount lustre-MDT0001 complete [ 9405.302278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9405.493940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9405.516793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9405.723303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9410.315893] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9419.063937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9419.225427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9419.278685] Lustre: 331861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9419.510982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9424.273059] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9431.936534] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9437.478044] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9447.738433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.936915] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9447.946757] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9447.956304] LustreError: 333749:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9447.961787] LustreError: 333749:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9447.966410] LustreError: 333749:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9448.026440] Lustre: server umount lustre-OST0000 complete [ 9448.028361] LustreError: 333749:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9458.681668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9458.951213] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9459.164043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9465.718961] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9467.071967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9473.670743] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9481.736604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9481.886127] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9482.001515] Lustre: lustre-OST0001: new disk, initializing [ 9482.010388] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9482.026241] Lustre: 335288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9482.170381] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9489.044169] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9489.546431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9489.554874] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9489.685178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9496.667465] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9500.141360] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9500.148313] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9500.153430] LustreError: Skipped 1 previous similar message [ 9500.168070] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9500.175783] Lustre: Skipped 1 previous similar message [ 9505.269031] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9505.277306] Lustre: Skipped 1 previous similar message [ 9505.914905] Lustre: server umount lustre-OST0001 complete [ 9510.375952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9510.377378] LustreError: 334216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.386145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9510.388781] Lustre: Skipped 2 previous similar messages [ 9510.406400] LustreError: 334216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9515.521984] LustreError: 334217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9515.550407] LustreError: 334217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9516.588241] Lustre: server umount lustre-OST0000 complete [ 9520.610675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9520.614202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9520.636181] Lustre: Skipped 1 previous similar message [ 9520.661995] Lustre: Skipped 4 previous similar messages [ 9526.649116] Lustre: server umount lustre-MDT0000 complete [ 9530.160204] LustreError: 331899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835684 with bad export cookie 4489619061713341428 [ 9530.161605] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9530.220780] LustreError: 331899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9530.537365] Lustre: server umount lustre-MDT0001 complete [ 9542.594993] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [ 9545.580719] Key type lgssc unregistered [ 9546.029838] LNet: 337263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9546.055362] LNetError: 337263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9547.112127] LNet: Removed LNI 192.168.206.133@tcp [ 9547.948572] Key type .llcrypt unregistered [ 9547.951333] Key type ._llcrypt unregistered [ 9617.855483] Key type ._llcrypt registered [ 9617.859181] Key type .llcrypt registered [ 9617.950307] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [ 9634.784271] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [ 9636.927396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9637.014708] alg: No test for adler32 (adler32-zlib) [ 9638.208168] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 9638.576444] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [ 9640.328175] Key type lgssc registered [ 9641.401467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9676.205536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9676.249879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9677.574895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9677.624374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9677.731821] Lustre: lustre-MDT0000: new disk, initializing [ 9677.958589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9678.011768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9682.945806] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9695.863536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9696.142734] Lustre: 342196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9696.247778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9696.256442] Lustre: Skipped 1 previous similar message [ 9696.390116] Lustre: lustre-MDT0001: new disk, initializing [ 9696.603382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9696.746257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9696.770773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9701.765500] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9711.986780] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9718.271266] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9725.247830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9725.522383] Lustre: lustre-OST0000: new disk, initializing [ 9725.527897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9725.536311] Lustre: 344165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9725.624265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9731.283857] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9731.635564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9731.650173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9731.724054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9740.208887] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9744.521356] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9744.729601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9748.697192] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9748.912639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9752.035605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.036648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9752.054730] Lustre: Skipped 1 previous similar message [ 9756.801857] Lustre: server umount lustre-OST0000 complete [ 9762.276399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.284049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9762.297468] Lustre: Skipped 1 previous similar message [ 9762.313222] Lustre: Skipped 2 previous similar messages [ 9767.374154] Lustre: server umount lustre-MDT0000 complete [ 9771.016805] LustreError: 342189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835924 with bad export cookie 6190271433756169113 [ 9771.022745] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9771.032305] LustreError: 342189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9771.481498] Lustre: server umount lustre-MDT0001 complete [ 9780.301583] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:05:32 (1787835932) [ 9787.575901] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:05:39 (1787835939) [ 9795.053686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9795.396730] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9795.412907] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9795.460582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9799.548858] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9800.677568] LustreError: 346921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9805.794226] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.332408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9808.599855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9813.625792] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9823.902858] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9831.746996] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9840.164838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9840.725855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9848.558084] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9857.872598] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9865.405803] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9867.422219] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9869.298039] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9871.473933] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9873.303121] Lustre: Failing over lustre-MDT0000 [ 9873.557306] Lustre: server umount lustre-MDT0000 complete [ 9874.411396] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9874.425597] LustreError: 347649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.434923] Lustre: Skipped 2 previous similar messages [ 9874.462259] LustreError: 347649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9879.524286] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.539527] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9889.766694] LustreError: 347649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9889.798316] LustreError: 347649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9890.786785] Lustre: 340423:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836028/real 1787836028] req@000000002301ae27 x1874681703503488/t0(0) o400->MGC192.168.206.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787836044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9890.832070] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9891.736973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.028089] LustreError: 340420:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007094f35a x1874681703509248/t0(0) o250->MGC192.168.206.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9901.418716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.491408] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9905.737552] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9906.512230] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9906.673044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9906.755042] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9906.800341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9916.724301] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9918.844172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9922.386702] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9924.696168] Lustre: Failing over lustre-MDT0000 [ 9925.055628] Lustre: server umount lustre-MDT0000 complete [ 9927.034336] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9927.061401] LustreError: 346917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 9927.147196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9927.156524] Lustre: Skipped 1 previous similar message [ 9943.466885] Lustre: 340423:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836081/real 1787836081] req@00000000432f72bb x1874681703524608/t0(0) o400->MGC192.168.206.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787836097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9943.524949] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9943.713318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.769841] LustreError: 340420:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000385f9e7f x1874681703530752/t0(0) o250->MGC192.168.206.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9954.115802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9954.165778] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9957.724502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9958.748621] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [ 9959.410317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9959.417362] Lustre: Skipped 2 previous similar messages [ 9959.446675] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9959.507623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9968.945812] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9970.424528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9979.877043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9979.880620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.895721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9979.924358] Lustre: Skipped 3 previous similar messages [ 9983.232813] Lustre: server umount lustre-MDT0000 complete [ 9984.994032] LustreError: 346921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.011838] LustreError: 346921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 9987.480669] LustreError: 346902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836141 with bad export cookie 6190271433756171829 [ 9987.499490] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9987.506267] LustreError: 346902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9988.098109] Lustre: server umount lustre-MDT0001 complete [10003.751110] Lustre: server umount lustre-OST0000 complete [10022.198584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.792339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10027.343273] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10037.620415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10042.795687] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10052.306388] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10058.707464] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10067.488118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10073.798914] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10081.262448] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10092.477615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10102.753226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10102.757728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10102.760349] LustreError: Skipped 1 previous similar message [10102.761408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10102.779946] Lustre: Skipped 1 previous similar message [10102.794813] Lustre: Skipped 3 previous similar messages [10107.590748] Lustre: server umount lustre-OST0000 complete [10112.996034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10113.005072] Lustre: Skipped 1 previous similar message [10122.210318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10122.215111] Lustre: Skipped 3 previous similar messages [10125.280113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10125.429493] Lustre: server umount lustre-MDT0000 complete [10127.330057] LustreError: 354556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10127.351969] LustreError: 354556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10128.929538] LustreError: 353246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836282 with bad export cookie 6190271433756172571 [10128.939053] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10128.941776] LustreError: 353246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10129.190538] Lustre: server umount lustre-MDT0001 complete [10139.304618] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [10141.866858] Key type lgssc unregistered [10142.138757] LNet: 356824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10142.154939] LNetError: 356824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10143.207211] LNet: Removed LNI 192.168.206.133@tcp [10144.012942] Key type .llcrypt unregistered [10144.014888] Key type ._llcrypt unregistered [10167.365933] Key type ._llcrypt registered [10167.367810] Key type .llcrypt registered [10167.477509] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10168.338516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10168.549073] alg: No test for adler32 (adler32-zlib) [10169.597119] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [10169.775959] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [10171.441675] Key type lgssc registered [10172.278160] Lustre: Echo OBD driver; http://www.lustre.org/ [10179.292535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10179.308980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.603739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10184.919966] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10190.822158] LustreError: 358354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10190.845705] LustreError: 358354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10194.407498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10194.803920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10199.342559] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10206.510346] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10212.517296] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10220.009279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10220.228548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10225.354739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10226.762324] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10236.071791] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10256.354900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10256.361477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10256.369866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10256.396980] Lustre: Skipped 1 previous similar message [10258.006425] Lustre: server umount lustre-MDT0000 complete [10261.484338] LustreError: 358354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.506107] LustreError: 358354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10262.212186] LustreError: 359089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836416 with bad export cookie 16074097942766612158 [10262.213324] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10262.228268] LustreError: 359089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10262.677500] Lustre: server umount lustre-MDT0001 complete [10277.054536] Lustre: server umount lustre-OST0000 complete [10292.499136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10292.988530] LustreError: 362088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10293.063469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10298.340597] LustreError: 362089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10298.363300] LustreError: 362089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10298.928232] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10302.439313] LustreError: 362088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10308.629337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.975698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10313.545347] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10321.767694] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10328.366167] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10336.055536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10336.474908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10337.544489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10343.657763] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10351.963601] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10364.389509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10364.395496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10364.407544] Lustre: Skipped 3 previous similar messages [10364.417706] Lustre: Skipped 4 previous similar messages [10369.512292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10369.521130] Lustre: Skipped 2 previous similar messages [10370.658470] Lustre: server umount lustre-MDT0000 complete [10374.631549] LustreError: 362818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10374.658745] LustreError: 362818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10374.852240] LustreError: 362070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836528 with bad export cookie 16074097942766613278 [10374.860522] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10374.867796] LustreError: 362070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10375.387747] Lustre: server umount lustre-MDT0001 complete [10389.986762] Lustre: server umount lustre-OST0000 complete [10405.035988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10405.542745] LustreError: 366000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.640430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10409.756035] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10418.030151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.598117] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10429.253711] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10434.642959] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10440.641643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10440.938272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10440.944494] Lustre: Skipped 1 previous similar message [10447.017879] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10450.168103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10454.704799] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10459.619904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10459.625841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10459.637446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10460.645468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10465.802379] Lustre: server umount lustre-OST0000 complete [10469.857070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10469.870055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10469.883337] Lustre: Skipped 1 previous similar message [10469.896205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10469.905895] Lustre: Skipped 2 previous similar messages [10475.957609] Lustre: server umount lustre-MDT0000 complete [10476.006254] LustreError: 366000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10476.025988] LustreError: 366000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10479.604755] LustreError: 365981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836633 with bad export cookie 16074097942766614300 [10479.615028] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10479.618805] LustreError: 365981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10479.900585] Lustre: server umount lustre-MDT0001 complete [10490.197472] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [10492.633630] Key type lgssc unregistered [10492.923339] LNet: 369534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10492.929789] LNetError: 369534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10492.949158] LNet: Removed LNI 192.168.206.133@tcp [10493.722861] Key type .llcrypt unregistered [10493.728193] Key type ._llcrypt unregistered [10517.275300] Key type ._llcrypt registered [10517.277530] Key type .llcrypt registered [10517.386717] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10518.798733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10518.844715] alg: No test for adler32 (adler32-zlib) [10520.086952] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [10520.279724] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [10521.984102] Key type lgssc registered [10523.333329] Lustre: Echo OBD driver; http://www.lustre.org/ [10531.189771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10531.212422] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10531.218180] LustreError: 370973:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10531.232232] LustreError: 370973:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10531.238717] LustreError: 370973:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10531.242437] LustreError: 370973:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10531.247449] LustreError: 370973:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10538.433480] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10538.437836] LustreError: 371370:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10538.451468] LustreError: 371370:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10538.460084] LustreError: 371370:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10538.465923] LustreError: 371370:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10538.471966] LustreError: 371370:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10557.389375] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [10559.950707] Key type lgssc unregistered [10560.283932] LNet: 372377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10560.298775] LNetError: 372377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10560.315884] LNet: Removed LNI 192.168.206.133@tcp [10561.196889] Key type .llcrypt unregistered [10561.201651] Key type ._llcrypt unregistered [10578.297331] Key type ._llcrypt registered [10578.299453] Key type .llcrypt registered [10578.389713] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [10592.778382] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10593.777293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10593.969650] alg: No test for adler32 (adler32-zlib) [10595.003370] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [10595.203397] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [10596.890077] Key type lgssc registered [10598.131756] Lustre: Echo OBD driver; http://www.lustre.org/ [10635.460446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10635.508074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.885876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10636.922893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10637.042277] Lustre: lustre-MDT0000: new disk, initializing [10637.135408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10637.148142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10642.029860] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10652.431774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10652.516952] Lustre: 375540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10652.584985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10652.588412] Lustre: Skipped 1 previous similar message [10652.671657] Lustre: lustre-MDT0001: new disk, initializing [10652.751796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10652.799028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10652.830937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10657.713606] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10668.969413] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10676.605293] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10683.657694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10684.057453] Lustre: lustre-OST0000: new disk, initializing [10684.064699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10684.081556] Lustre: 377513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10684.196693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10691.348545] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10691.684223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10691.707255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10691.845038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10701.066204] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10705.575615] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10705.886643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10711.069805] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10711.293505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10717.668359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.670711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10717.698933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10719.743112] Lustre: server umount lustre-OST0000 complete [10724.323990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10724.333895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10724.349298] Lustre: Skipped 1 previous similar message [10724.359452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10724.365559] Lustre: Skipped 1 previous similar message [10727.906260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10727.909718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10727.947643] Lustre: Skipped 1 previous similar message [10730.644330] Lustre: server umount lustre-MDT0000 complete [10735.477455] LustreError: 377512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836889 with bad export cookie 15676788083338579742 [10735.498408] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10735.501762] LustreError: 377512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10735.871155] Lustre: server umount lustre-MDT0001 complete [10745.258496] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10747.104378] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:21:39 (1787836899) [10755.176611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10756.059943] LustreError: 379973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10756.090052] LustreError: 379973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10756.250480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10761.698979] LustreError: 379974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10761.878633] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10765.795642] LustreError: 379973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10770.914225] LustreError: 379974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10772.280566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10773.003895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10777.100657] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10784.161422] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10789.677576] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10796.237649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10796.684857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10803.088471] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10810.682212] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10817.966677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10818.132689] Lustre: lustre-OST0001: new disk, initializing [10818.136192] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10818.141604] Lustre: 382810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10818.223755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10824.077455] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10827.908737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10827.919173] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10828.022394] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10834.490775] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10842.483985] Lustre: lustre-OST0001: Not available for connect from 192.168.206.33@tcp (stopping) [10843.618572] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10843.624368] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10843.628777] LustreError: Skipped 1 previous similar message [10843.643526] Lustre: Skipped 1 previous similar message [10845.164741] Lustre: server umount lustre-OST0001 complete [10847.559700] LustreError: 381790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10858.986917] LustreError: 383734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10859.004598] LustreError: 383734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10863.589187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10863.598062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10863.627611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10863.634765] Lustre: Skipped 2 previous similar messages [10867.787361] Lustre: server umount lustre-OST0000 complete [10874.339353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10874.364254] Lustre: Skipped 1 previous similar message [10874.368706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10874.376970] Lustre: Skipped 2 previous similar messages [10878.232112] Lustre: server umount lustre-MDT0000 complete [10879.458163] LustreError: 379974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10879.486825] LustreError: 379974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10882.477252] LustreError: 379954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837036 with bad export cookie 15676788083338580631 [10882.481394] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10882.488338] LustreError: 379954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10889.198504] Lustre: server umount lustre-MDT0001 complete [10903.243515] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [10906.949888] Key type lgssc unregistered [10907.358573] LNet: 384801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10907.369153] LNetError: 384801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10907.385715] LNet: Removed LNI 192.168.206.133@tcp [10908.601391] Key type .llcrypt unregistered [10908.610750] Key type ._llcrypt unregistered [10961.654386] Key type ._llcrypt registered [10961.657763] Key type .llcrypt registered [10961.753930] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [10962.563346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10962.667698] alg: No test for adler32 (adler32-zlib) [10963.666567] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [10963.895152] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [10965.576165] Key type lgssc registered [10966.672750] Lustre: Echo OBD driver; http://www.lustre.org/ [10974.007950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10974.039611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.356803] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10975.393777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10975.551356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10979.715631] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [10988.970186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.060171] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10989.092773] Lustre: 387603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10989.443341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10993.556770] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11002.226986] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11008.302418] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11015.138915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11015.140866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11015.168375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.186855] Lustre: Skipped 1 previous similar message [11016.387462] Lustre: server umount lustre-MDT0000 complete [11020.126425] LustreError: 387596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837173 with bad export cookie 6461317296359792402 [11020.128225] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11020.133854] LustreError: 387596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11020.327315] Lustre: server umount lustre-MDT0001 complete [11029.683405] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:26:21 (1787837181) [11036.697717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11037.262181] LustreError: 390115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11037.277615] LustreError: 390115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11037.314314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11041.992867] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11042.795225] LustreError: 390116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11047.906087] LustreError: 390115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.790874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.143903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11054.515697] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11061.865908] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11067.633289] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11074.119588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11074.356776] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11074.597633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11080.622316] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11088.492743] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11097.482380] Lustre: Permanently deactivating lustre-OST0000 [11097.493487] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11103.294730] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11104.557842] LustreError: 393052:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11104.570806] LustreError: 393052:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11104.584197] LustreError: 393052:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11107.298777] LustreError: 393101:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11107.304707] LustreError: 393101:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11115.502133] Lustre: server umount lustre-OST0000 complete [11117.947708] LustreError: 393350:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11117.955465] LustreError: 393350:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11121.633374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11121.643573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11121.659369] Lustre: Skipped 1 previous similar message [11121.668438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11122.658822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11122.669087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11127.777079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11127.786297] Lustre: Skipped 2 previous similar messages [11129.927173] Lustre: server umount lustre-MDT0000 complete [11132.900766] LustreError: 390116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11132.936107] LustreError: 390116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11134.043925] LustreError: 391959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837287 with bad export cookie 6461317296359792983 [11134.057053] LustreError: 391959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11134.057478] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11143.879393] Lustre: server umount lustre-MDT0001 complete [11151.286555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11156.516743] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11159.659964] LustreError: 394509:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11165.823396] Lustre: 394848:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.133@tcp [11167.760314] Lustre: server umount MGS complete [11179.377959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11179.859693] LustreError: 395572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11179.889379] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11179.897507] Lustre: Skipped 1 previous similar message [11179.954792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11184.144582] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11190.241777] LustreError: 395573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11190.261779] LustreError: 395573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11192.349731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.612217] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11197.164729] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11204.402890] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11209.980548] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11211.593781] Lustre: Permanently reactivating lustre-OST0000 [11211.611699] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11211.615972] Lustre: Skipped 2 previous similar messages [11238.886377] LustreError: 397351:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11245.605948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.917906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11245.925876] Lustre: Skipped 1 previous similar message [11251.342164] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11258.598031] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11264.482227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11264.485177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11264.496688] LustreError: Skipped 1 previous similar message [11264.503257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11264.518073] Lustre: Skipped 1 previous similar message [11269.758856] Lustre: server umount lustre-OST0000 complete [11274.722864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11274.726377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11274.744231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11274.758567] Lustre: Skipped 4 previous similar messages [11279.832139] Lustre: server umount lustre-MDT0000 complete [11279.849098] LustreError: 395572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11279.872147] LustreError: 395572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11283.078738] LustreError: 395553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837436 with bad export cookie 6461317296359794033 [11283.083983] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11283.086697] LustreError: 395553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11283.378043] Lustre: server umount lustre-MDT0001 complete [11292.998397] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [11295.613993] Key type lgssc unregistered [11295.912465] LNet: 399989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11295.918489] LNetError: 399989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11295.941852] LNet: Removed LNI 192.168.206.133@tcp [11296.790604] Key type .llcrypt unregistered [11296.795622] Key type ._llcrypt unregistered [11313.816156] Key type ._llcrypt registered [11313.819200] Key type .llcrypt registered [11313.936517] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [11331.340685] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [11333.285456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11333.339406] alg: No test for adler32 (adler32-zlib) [11334.488607] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [11334.692615] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [11336.360405] Key type lgssc registered [11337.743257] Lustre: Echo OBD driver; http://www.lustre.org/ [11369.961696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11369.984319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11371.265501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11371.301819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11371.372949] Lustre: lustre-MDT0000: new disk, initializing [11371.463331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11371.480883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11375.752297] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11385.680463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11385.775434] Lustre: 403143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11385.830468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11385.834672] Lustre: Skipped 1 previous similar message [11385.917956] Lustre: lustre-MDT0001: new disk, initializing [11385.998631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11386.044673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11386.061202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11390.142944] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11398.016601] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11402.851245] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11409.616498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11409.912119] Lustre: lustre-OST0000: new disk, initializing [11409.917137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11409.926448] Lustre: 405114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11410.022594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11416.315650] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11417.162972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11417.179232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11417.279957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11424.684208] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11429.237185] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11429.423982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11433.860849] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11434.094874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11438.049757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11438.057141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11438.059236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11438.074875] Lustre: Skipped 1 previous similar message [11438.082654] Lustre: Skipped 1 previous similar message [11441.832631] Lustre: server umount lustre-OST0000 complete [11447.777570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11447.782361] LustreError: Skipped 1 previous similar message [11447.786629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.801062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11451.342139] Lustre: server umount lustre-MDT0000 complete [11454.718043] LustreError: 403137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837608 with bad export cookie 1330617636665833212 [11454.725421] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11454.733682] LustreError: 403137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11455.047707] Lustre: server umount lustre-MDT0001 complete [11463.872850] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11465.410653] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:33:37 (1787837617) [11471.660061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.150908] LustreError: 407571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.176456] LustreError: 407571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11472.231036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11477.129439] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11477.474875] LustreError: 407572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.595178] LustreError: 407571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11485.153559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11485.504897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11489.646834] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11496.412962] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11501.606206] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11507.657517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11507.974150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11513.798641] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11520.401373] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11522.172708] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11524.627455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11531.746641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11531.757923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11531.777243] Lustre: Skipped 1 previous similar message [11531.781491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11531.785959] Lustre: Skipped 2 previous similar messages [11535.145982] Lustre: server umount lustre-OST0000 complete [11541.994455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11542.009151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11542.022939] Lustre: Skipped 1 previous similar message [11542.031040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11542.036497] Lustre: Skipped 2 previous similar messages [11545.390291] Lustre: server umount lustre-MDT0000 complete [11548.130598] LustreError: 407572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11548.147925] LustreError: 407572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11548.664032] LustreError: 407553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837702 with bad export cookie 1330617636665834094 [11548.672754] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11548.677562] LustreError: 407553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11549.056564] Lustre: server umount lustre-MDT0001 complete [11559.547371] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [11562.522439] Key type lgssc unregistered [11562.814408] LNet: 411298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11562.819906] LNetError: 411298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11563.879545] LNet: Removed LNI 192.168.206.133@tcp [11564.717994] Key type .llcrypt unregistered [11564.726296] Key type ._llcrypt unregistered [11590.591761] Key type ._llcrypt registered [11590.594736] Key type .llcrypt registered [11590.694895] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [11590.989380] Key type .llcrypt unregistered [11590.996303] Key type ._llcrypt unregistered [11608.706825] Key type ._llcrypt registered [11608.709330] Key type .llcrypt registered [11608.790909] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [11609.726813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11609.880063] alg: No test for adler32 (adler32-zlib) [11610.962715] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [11611.199594] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [11612.960497] Key type lgssc registered [11614.162030] Lustre: Echo OBD driver; http://www.lustre.org/ [11621.243765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11621.275736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11622.668550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11627.480548] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11633.121822] LustreError: 413519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.138953] LustreError: 413519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11636.157112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11636.429925] LustreError: 413518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11636.451920] LustreError: 413518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11636.467650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11641.828692] LustreError: 413513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.851025] LustreError: 413513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11642.099643] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11646.956958] LustreError: 413514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11646.982195] LustreError: 413514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11647.990647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11649.855507] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11653.107987] LustreError: 414861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11653.119116] LustreError: 414861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11656.779990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11657.244940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11661.851674] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11672.324787] Lustre: server umount lustre-OST0000 complete [11677.666984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11677.667057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11677.678712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11677.692290] Lustre: Skipped 1 previous similar message [11677.702447] Lustre: Skipped 1 previous similar message [11682.785733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.788617] Lustre: Skipped 1 previous similar message [11685.981439] Lustre: server umount lustre-MDT0000 complete [11687.910968] LustreError: 415253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11687.932816] LustreError: 415253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11688.928855] Lustre: 412831:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837826/real 1787837826] req@0000000015e1ae5a x1874683772299776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787837842 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11688.939375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11688.964661] Lustre: 412831:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11689.359845] LustreError: 413498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837843 with bad export cookie 10332516886185999637 [11689.366540] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11689.376752] LustreError: 413498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11689.680014] Lustre: server umount lustre-MDT0001 complete [11700.133806] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [11702.809557] Key type lgssc unregistered [11703.153507] LNet: 416668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11703.181160] LNetError: 416668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11703.197894] LNet: Removed LNI 192.168.206.133@tcp [11704.005411] Key type .llcrypt unregistered [11704.007365] Key type ._llcrypt unregistered [11726.916675] Key type ._llcrypt registered [11726.920789] Key type .llcrypt registered [11727.002389] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [11727.924979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11728.118068] alg: No test for adler32 (adler32-zlib) [11729.145836] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [11729.404703] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [11731.080460] Key type lgssc registered [11732.244730] Lustre: Echo OBD driver; http://www.lustre.org/ [11738.804258] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11738.831338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11754.464672] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11759.584522] LustreError: 418062:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.133@tcp: failed processing log, type 4: rc = -110 [11786.336249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11792.336542] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11799.849812] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11806.185257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11806.290184] LustreError: 417398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007795aebc x1874683896006912/t0(0) o250->MGC192.168.206.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11806.635915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11806.687785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11810.824298] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11811.813745] LustreError: 418088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.842487] LustreError: 418088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11816.933970] LustreError: 418087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.083156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11820.678371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11825.060667] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11832.863661] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11837.930634] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11846.116995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11846.125428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11846.143683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11846.906645] Lustre: server umount lustre-OST0000 complete [11851.234304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11851.245619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11851.263315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11852.777323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11852.779063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11857.890619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11857.906162] Lustre: Skipped 2 previous similar messages [11861.068812] Lustre: server umount lustre-MDT0000 complete [11863.016083] LustreError: 419103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.644442] LustreError: 418070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838019 with bad export cookie 16484345682312323938 [11865.645416] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11865.658474] LustreError: 418070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11866.063693] Lustre: server umount lustre-MDT0001 complete [11876.685604] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [11879.638286] Key type lgssc unregistered [11879.938525] LNet: 421637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11879.946225] LNetError: 421637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11879.964433] LNet: Removed LNI 192.168.206.133@tcp [11881.150500] Key type .llcrypt unregistered [11881.152230] Key type ._llcrypt unregistered [11902.659947] Key type ._llcrypt registered [11902.662342] Key type .llcrypt registered [11902.748414] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [11903.795594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11903.905905] alg: No test for adler32 (adler32-zlib) [11905.022161] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [11905.299742] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [11906.976448] Key type lgssc registered [11908.386201] Lustre: Echo OBD driver; http://www.lustre.org/ [11917.186449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11917.221832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.922033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11923.500255] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11929.060989] LustreError: 423055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.089313] LustreError: 423055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11931.474763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.850330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11935.323831] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11942.596778] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11948.348847] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11953.654338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11953.918503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11959.307564] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [11964.176544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11966.171727] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11973.094849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11973.101598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11973.111566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11973.989694] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.33@tcp (stopping) [11974.631577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11974.661298] Lustre: Skipped 1 previous similar message [11977.264355] Lustre: server umount lustre-MDT0000 complete [11979.085776] LustreError: 423051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.796387] LustreError: 423035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.33@tcp arrived at 1787838133 with bad export cookie 811737557739416941 [11979.822639] LustreError: 423035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11981.728383] LustreError: 423037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838135 with bad export cookie 811737557739416045 [11981.733751] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11981.741295] LustreError: 423037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11982.817551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11982.825460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11982.855964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11982.868314] Lustre: Skipped 3 previous similar messages [11984.865094] LustreError: 423056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11984.872707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.887196] LustreError: 423056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11986.917065] LustreError: 423050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.930153] LustreError: 423050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11987.970524] Lustre: server umount lustre-OST0000 complete [11992.038725] LustreError: 424154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11992.059376] LustreError: 424154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11994.089654] Lustre: server umount lustre-MDT0001 complete [12003.993459] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [12006.501383] Key type lgssc unregistered [12006.742720] LNet: 426740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12006.749889] LNetError: 426740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12006.775825] LNet: Removed LNI 192.168.206.133@tcp [12007.743494] Key type .llcrypt unregistered [12007.746853] Key type ._llcrypt unregistered [12032.092473] Key type ._llcrypt registered [12032.097815] Key type .llcrypt registered [12032.359694] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [12033.763216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12033.913141] alg: No test for adler32 (adler32-zlib) [12034.967262] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12035.342863] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [12037.064248] Key type lgssc registered [12038.655510] Lustre: Echo OBD driver; http://www.lustre.org/ [12046.813806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12046.823048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12048.266660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12052.983209] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12058.597243] LustreError: 428159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12058.619240] LustreError: 428159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12061.312289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12061.754022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12066.043715] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12073.670431] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12079.490918] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12086.799708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12087.232182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12093.357380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12093.753990] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12102.050985] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12107.745928] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12107.752325] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12107.762619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12108.769316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12108.776114] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12110.163435] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.33@tcp (stopping) [12110.179761] Lustre: Skipped 1 previous similar message [12112.663455] Lustre: server umount lustre-MDT0001 complete [12113.891856] LustreError: 428154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12113.921612] LustreError: 428154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12119.011824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.012157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12119.020993] LustreError: 428159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12119.041627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12124.129318] LustreError: 428158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12124.137351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12124.946726] Lustre: server umount lustre-OST0000 complete [12129.120843] Lustre: server umount lustre-MDT0000 complete [12143.880421] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [12147.052529] Key type lgssc unregistered [12147.335221] LNet: 431846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12147.345521] LNetError: 431846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12147.377414] LNet: Removed LNI 192.168.206.133@tcp [12148.260885] Key type .llcrypt unregistered [12148.265315] Key type ._llcrypt unregistered [12177.825035] Key type ._llcrypt registered [12177.826580] Key type .llcrypt registered [12177.890368] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [12193.391096] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [12194.584335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12194.600439] alg: No test for adler32 (adler32-zlib) [12195.605113] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12195.801287] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [12197.488228] Key type lgssc registered [12198.481131] Lustre: Echo OBD driver; http://www.lustre.org/ [12233.831714] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [12247.826605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12247.850519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12249.062471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12249.109760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12249.203442] Lustre: lustre-MDT0000: new disk, initializing [12249.371162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12249.385503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12253.329965] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12268.738674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12269.042432] Lustre: lustre-OST0000: new disk, initializing [12269.048248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12269.053698] Lustre: Skipped 1 previous similar message [12269.058231] Lustre: 436881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12269.118862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12275.312186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12275.318089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12275.361559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12275.657583] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12287.015866] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12305.425631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12305.606817] Lustre: 435864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12305.740085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12305.866196] Lustre: lustre-MDT0001: new disk, initializing [12306.052080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12306.133574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12306.151452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12310.176680] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12317.804387] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12318.005922] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12321.692884] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12321.915189] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12330.351735] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.33@tcp (stopping) [12331.495081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12331.497968] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12331.520126] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12331.529664] Lustre: Skipped 1 previous similar message [12332.969943] Lustre: server umount lustre-MDT0001 complete [12341.733347] LustreError: 439431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12341.756733] LustreError: 439431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12346.851406] LustreError: 435870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.888819] LustreError: 435870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12351.972659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12351.974389] LustreError: 439431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.975583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12351.975592] Lustre: Skipped 1 previous similar message [12351.976576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12357.097682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12357.099026] LustreError: 435876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12362.209816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12362.210786] LustreError: 435875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12364.256199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12364.395582] Lustre: server umount lustre-OST0000 complete [12382.176942] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12382.539277] Lustre: server umount lustre-MDT0000 complete [12397.429975] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [12401.036416] Key type lgssc unregistered [12401.333191] LNet: 440634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12401.340115] LNetError: 440634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12401.359654] LNet: Removed LNI 192.168.206.133@tcp [12402.276716] Key type .llcrypt unregistered [12402.279215] Key type ._llcrypt unregistered [12439.525462] Key type ._llcrypt registered [12439.527101] Key type .llcrypt registered [12439.696473] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [12440.014302] Key type .llcrypt unregistered [12440.016495] Key type ._llcrypt unregistered [12459.142305] Key type ._llcrypt registered [12459.144385] Key type .llcrypt registered [12459.226635] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [12460.692716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12460.724917] alg: No test for adler32 (adler32-zlib) [12461.939408] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12462.202660] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [12463.920176] Key type lgssc registered [12464.913813] Lustre: Echo OBD driver; http://www.lustre.org/ [12492.394870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12492.441820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12493.697397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12493.735097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12493.847916] Lustre: lustre-MDT0000: new disk, initializing [12493.954947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12493.982192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12498.790428] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12509.150110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.365759] Lustre: 444030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12509.440810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12509.443850] Lustre: Skipped 1 previous similar message [12509.571842] Lustre: lustre-MDT0001: new disk, initializing [12509.712917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12509.776529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12509.795138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12514.863079] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12524.394257] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12530.679023] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12538.077684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12538.438072] Lustre: lustre-OST0000: new disk, initializing [12538.444876] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12538.454452] Lustre: 446001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12538.548680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12545.267164] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12546.091142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12546.099827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12546.176957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12556.964889] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12565.988879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12566.025832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12566.046961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12571.394535] Lustre: server umount lustre-MDT0000 complete [12575.481483] LustreError: 446000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838729 with bad export cookie 7436938136436734536 [12575.486776] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12575.492691] LustreError: 446000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12575.759714] Lustre: server umount lustre-MDT0001 complete [12590.723748] Lustre: server umount lustre-OST0000 complete [12602.950439] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:52:35 (1787838755) [12611.064547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12611.529679] LustreError: 448029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.546996] LustreError: 448029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12611.595917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12615.975341] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12616.674537] LustreError: 448030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12621.793403] LustreError: 448029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.050725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.477502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12630.135630] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12637.568349] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12644.170862] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12664.235831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12664.528686] Lustre: Found index 0 for lustre-OST0000, updating log [12664.756860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12670.619550] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [12679.190174] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12683.209784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12691.427056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12691.436948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12691.458360] Lustre: Skipped 2 previous similar messages [12691.466940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12691.475738] Lustre: Skipped 3 previous similar messages [12692.519943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12692.530951] Lustre: Skipped 2 previous similar messages [12695.875765] Lustre: server umount lustre-OST0000 complete [12712.421756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12712.433310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12712.447217] Lustre: Skipped 1 previous similar message [12712.451821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12718.353367] Lustre: server umount lustre-MDT0000 complete [12719.081657] LustreError: 448757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12719.099174] LustreError: 448757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12722.113351] LustreError: 448010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838875 with bad export cookie 7436938136436735929 [12722.115135] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12722.122546] LustreError: 448010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12722.557522] Lustre: server umount lustre-MDT0001 complete [12725.412816] LNet: 1 local NIs in recovery (showing 1): 192.168.206.133@tcp [12732.351846] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:54:44 (1787838884) [13350.032593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13350.507329] LustreError: 454455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13350.522348] LustreError: 454455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13350.602704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13354.485191] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [13356.008635] LustreError: 454456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13361.121858] LustreError: 454455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13362.228613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13362.503542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13366.910182] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [13373.303772] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13377.792162] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13397.552435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13397.839367] Lustre: Found index 0 for lustre-OST0000, updating log [13398.147973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13400.311401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13403.360124] LNet: 1 local NIs in recovery (showing 1): 192.168.206.133@tcp [13404.119656] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [13411.242948] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13414.338375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13414.357834] Lustre: Skipped 1 previous similar message [13414.375749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13414.383063] Lustre: Skipped 2 previous similar messages [13419.598496] Lustre: server umount lustre-OST0000 complete [13424.100302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13424.112427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13424.128943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13425.187308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13425.192943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13427.298110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13427.311356] Lustre: Skipped 4 previous similar messages [13430.021237] Lustre: server umount lustre-MDT0000 complete [13430.382189] LustreError: 455703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13430.403877] LustreError: 455703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13434.380560] LustreError: 456311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839588 with bad export cookie 7436938136436737014 [13434.387454] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13434.395614] LustreError: 456311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13434.767370] Lustre: server umount lustre-MDT0001 complete [13464.708749] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [13467.872668] LNet: 1 local NIs in recovery (showing 1): 192.168.206.133@tcp [13473.986684] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [13507.200206] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:07:39 (1787839659) [13524.361597] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [13527.186694] Key type lgssc unregistered [13527.533141] LNet: 461074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13527.540679] LNetError: 461074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13657.640802] LNet: Removed LNI 192.168.206.133@tcp [13658.535507] Key type .llcrypt unregistered [13658.537282] Key type ._llcrypt unregistered [13674.799173] Key type ._llcrypt registered [13674.805600] Key type .llcrypt registered [13674.921470] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [13676.577272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13676.616429] alg: No test for adler32 (adler32-zlib) [13677.613535] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [13677.860811] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [13679.496160] Key type lgssc registered [13680.730778] Lustre: Echo OBD driver; http://www.lustre.org/ [13685.093832] LNet: Added LNI 192.168.206.133@tcp100 [8/256/0/180] [13685.926779] LNet: Added LNI 192.168.206.133@tcp101 [8/256/0/180] [13686.854240] LNet: Added LNI 192.168.206.133@tcp102 [8/256/0/180] [13687.882348] LNet: Added LNI 192.168.206.133@tcp103 [8/256/0/180] [13688.801686] LNet: Added LNI 192.168.206.133@tcp104 [8/256/0/180] [13689.813765] LNet: Added LNI 192.168.206.133@tcp105 [8/256/0/180] [13690.841199] LNet: Added LNI 192.168.206.133@tcp106 [8/256/0/180] [13691.780523] LNet: Added LNI 192.168.206.133@tcp107 [8/256/0/180] [13692.638366] LNet: Added LNI 192.168.206.133@tcp108 [8/256/0/180] [13693.573263] LNet: Added LNI 192.168.206.133@tcp109 [8/256/0/180] [13694.630314] LNet: Added LNI 192.168.206.133@tcp110 [8/256/0/180] [13695.924462] LNet: Added LNI 192.168.206.133@tcp111 [8/256/0/180] [13696.957635] LNet: Added LNI 192.168.206.133@tcp112 [8/256/0/180] [13697.853639] LNet: Added LNI 192.168.206.133@tcp113 [8/256/0/180] [13698.676727] LNet: Added LNI 192.168.206.133@tcp114 [8/256/0/180] [13699.574574] LNet: Added LNI 192.168.206.133@tcp115 [8/256/0/180] [13700.418771] LNet: Added LNI 192.168.206.133@tcp116 [8/256/0/180] [13701.167772] LNet: Added LNI 192.168.206.133@tcp117 [8/256/0/180] [13701.988171] LNet: Added LNI 192.168.206.133@tcp118 [8/256/0/180] [13702.801137] LNet: Added LNI 192.168.206.133@tcp119 [8/256/0/180] [13703.499383] LNet: Added LNI 192.168.206.133@tcp120 [8/256/0/180] [13704.424718] LNet: Added LNI 192.168.206.133@tcp121 [8/256/0/180] [13705.245480] LNet: Added LNI 192.168.206.133@tcp122 [8/256/0/180] [13706.504303] LNet: Added LNI 192.168.206.133@tcp123 [8/256/0/180] [13707.146088] LNet: Added LNI 192.168.206.133@tcp124 [8/256/0/180] [13707.902983] LNet: Added LNI 192.168.206.133@tcp125 [8/256/0/180] [13708.637254] LNet: Added LNI 192.168.206.133@tcp126 [8/256/0/180] [13709.449812] LNet: Added LNI 192.168.206.133@tcp127 [8/256/0/180] [13710.284565] LNet: Added LNI 192.168.206.133@tcp128 [8/256/0/180] [13711.476933] LNet: Added LNI 192.168.206.133@tcp129 [8/256/0/180] [13712.329620] LNet: Added LNI 192.168.206.133@tcp130 [8/256/0/180] [13713.100706] LNet: Added LNI 192.168.206.133@tcp131 [8/256/0/180] [13713.857496] LNet: Added LNI 192.168.206.133@tcp132 [8/256/0/180] [13714.612432] LNet: Added LNI 192.168.206.133@tcp133 [8/256/0/180] [13715.333985] LNet: Added LNI 192.168.206.133@tcp134 [8/256/0/180] [13716.115465] LNet: Added LNI 192.168.206.133@tcp135 [8/256/0/180] [13722.911715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13722.947439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13724.075017] Lustre: 464108:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.133@tcp: can't add NID 192.168.206.133@tcp130: rc = -75 [13724.130688] LNetError: 464132:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.133 [13724.134635] LNetError: Protocol error connecting to 192.168.206.133@tcp119 at host 192.168.206.133:988: is it running a compatible version of Lustre? [13724.138293] LNetError: Skipped 3 previous similar messages [13724.230637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13724.248918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13724.303630] Lustre: lustre-MDT0000: new disk, initializing [13724.353481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13724.364930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13724.960850] LNet: 5 local NIs in recovery (showing 5): 192.168.206.133@tcp125, 192.168.206.133@tcp126, 192.168.206.133@tcp129, 192.168.206.133@tcp128, 192.168.206.133@tcp127 [13725.152992] LNetError: 464132:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.133 [13725.190843] LNetError: 464132:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [13725.207483] LNetError: Protocol error connecting to 192.168.206.133@tcp at host 192.168.206.133:988: is it running a compatible version of Lustre? [13725.226546] LNetError: Skipped 2 previous similar messages [13727.010490] LNetError: 461567:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.133 [13727.026598] LNetError: 461567:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13727.049514] LNetError: Protocol error connecting to 192.168.206.133@tcp126 at host 192.168.206.133:988: is it running a compatible version of Lustre? [13727.061266] LNetError: Skipped 5 previous similar messages [13728.720811] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [13731.108335] LNetError: 461568:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.133 [13731.121404] LNetError: Protocol error connecting to 192.168.206.133@tcp126 at host 192.168.206.133:988: is it running a compatible version of Lustre? [13739.308338] LNetError: 461569:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.133 [13739.326948] LNetError: Protocol error connecting to 192.168.206.133@tcp126 at host 192.168.206.133:988: is it running a compatible version of Lustre? [13739.588750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13739.745087] Lustre: 464126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13739.840207] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13739.844037] Lustre: Skipped 1 previous similar message [13739.924860] Lustre: 464930:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.133@tcp: can't add NID 192.168.206.133@tcp130: rc = -75 [13739.931881] Lustre: lustre-MDT0001: new disk, initializing [13739.933127] Lustre: 464930:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13740.072953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13740.103381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13740.119866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13744.708579] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [13754.659782] LNetError: 464132:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.133 [13754.690279] LNetError: Protocol error connecting to 192.168.206.133@tcp124 at host 192.168.206.133:988: is it running a compatible version of Lustre? [13755.058555] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13761.228919] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13766.890031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13767.106924] Lustre: lustre-OST0000: new disk, initializing [13767.110236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13767.117532] Lustre: 466100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13767.217382] Lustre: 466105:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.133@tcp: can't add NID 192.168.206.133@tcp130: rc = -75 [13767.224723] Lustre: 466105:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [13767.289505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13772.864991] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [13773.089935] LNetError: 464129:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.133 [13773.095117] LNetError: 464129:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13773.100397] LNetError: Protocol error connecting to 192.168.206.133@tcp at host 192.168.206.133:988: is it running a compatible version of Lustre? [13773.104552] LNetError: Skipped 5 previous similar messages [13774.852878] Lustre: 466557:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.133@tcp: can't add NID 192.168.206.133@tcp130: rc = -75 [13774.868894] Lustre: 466557:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13774.918818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13774.927379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13775.032820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13780.256634] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13784.352435] LNet: 1 local NIs in recovery (showing 1): 192.168.206.133@tcp126 [13790.691236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13790.696888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13790.698524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13790.703446] LustreError: Skipped 1 previous similar message [13790.742382] Lustre: Skipped 1 previous similar message [13792.633277] Lustre: server umount lustre-OST0000 complete [13796.323972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13796.330968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13796.356459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13796.364091] Lustre: Skipped 1 previous similar message [13800.929607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13800.933136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13802.508475] Lustre: server umount lustre-MDT0000 complete [13805.752614] LustreError: 464118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839959 with bad export cookie 11805113824423508337 [13805.758730] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13805.774445] LustreError: 464118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13806.220727] Lustre: server umount lustre-MDT0001 complete [13813.026983] LNetError: 464133:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.133 [13813.036900] LNetError: 464133:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [13813.052907] LNetError: Protocol error connecting to 192.168.206.133@tcp at host 192.168.206.133:988: is it running a compatible version of Lustre? [13813.062360] LNetError: Skipped 6 previous similar messages [13817.114709] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [13819.684504] Key type lgssc unregistered [13819.984901] LNet: 468013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13819.998534] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13820.006848] LNet: Removed LNI 192.168.206.133@tcp [13821.024150] LNet: Removed LNI 192.168.206.133@tcp100 [13821.026979] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13821.034522] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13821.040407] LNet: Removed LNI 192.168.206.133@tcp101 [13822.048321] LNet: Removed LNI 192.168.206.133@tcp102 [13822.052227] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13822.065544] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13822.074765] LNet: Removed LNI 192.168.206.133@tcp103 [13823.136870] LNet: Removed LNI 192.168.206.133@tcp104 [13824.161463] LNet: Removed LNI 192.168.206.133@tcp105 [13824.169050] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13824.180210] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13824.185395] LNet: Removed LNI 192.168.206.133@tcp106 [13825.248327] LNet: Removed LNI 192.168.206.133@tcp107 [13826.272193] LNet: Removed LNI 192.168.206.133@tcp108 [13827.296200] LNet: Removed LNI 192.168.206.133@tcp109 [13828.320358] LNet: Removed LNI 192.168.206.133@tcp110 [13828.325646] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13828.335188] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13828.342979] LNet: Removed LNI 192.168.206.133@tcp111 [13829.408269] LNet: Removed LNI 192.168.206.133@tcp112 [13830.432148] LNet: Removed LNI 192.168.206.133@tcp113 [13831.456145] LNet: Removed LNI 192.168.206.133@tcp114 [13832.480411] LNet: Removed LNI 192.168.206.133@tcp115 [13833.504355] LNet: Removed LNI 192.168.206.133@tcp116 [13834.528788] LNet: Removed LNI 192.168.206.133@tcp117 [13835.552159] LNet: Removed LNI 192.168.206.133@tcp118 [13836.576948] LNet: Removed LNI 192.168.206.133@tcp119 [13836.579757] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13836.584683] LNetError: 468013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13836.589932] LNet: Removed LNI 192.168.206.133@tcp120 [13837.600161] LNet: Removed LNI 192.168.206.133@tcp121 [13838.624120] LNet: Removed LNI 192.168.206.133@tcp122 [13839.648545] LNet: Removed LNI 192.168.206.133@tcp123 [13840.673803] LNet: Removed LNI 192.168.206.133@tcp124 [13841.696270] LNet: Removed LNI 192.168.206.133@tcp125 [13841.702746] LNet: Removed LNI 192.168.206.133@tcp126 [13842.720182] LNet: Removed LNI 192.168.206.133@tcp127 [13843.744187] LNet: Removed LNI 192.168.206.133@tcp128 [13844.769088] LNet: Removed LNI 192.168.206.133@tcp129 [13844.772787] LNet: Removed LNI 192.168.206.133@tcp130 [13844.778877] LNet: Removed LNI 192.168.206.133@tcp131 [13844.781190] LNet: Removed LNI 192.168.206.133@tcp132 [13844.783490] LNet: Removed LNI 192.168.206.133@tcp133 [13844.785736] LNet: Removed LNI 192.168.206.133@tcp134 [13844.795867] LNet: Removed LNI 192.168.206.133@tcp135 [13845.288688] Key type .llcrypt unregistered [13845.291194] Key type ._llcrypt unregistered [13861.980601] Key type ._llcrypt registered [13861.983231] Key type .llcrypt registered [13862.052170] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [13875.176712] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [13876.055889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13876.210226] alg: No test for adler32 (adler32-zlib) [13877.207417] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [13877.471652] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [13879.112733] Key type lgssc registered [13880.002720] Lustre: Echo OBD driver; http://www.lustre.org/ [13914.705144] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:14:26 (1787840066) [13931.281598] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [13933.401578] Key type lgssc unregistered [13933.648360] LNet: 472104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13933.662756] LNetError: 472104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13933.689662] LNet: Removed LNI 192.168.206.133@tcp [13934.361016] Key type .llcrypt unregistered [13934.363639] Key type ._llcrypt unregistered [13949.096303] Key type ._llcrypt registered [13949.101233] Key type .llcrypt registered [13949.222374] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [13950.045867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13950.067560] alg: No test for adler32 (adler32-zlib) [13951.074633] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [13951.263396] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [13952.896223] Key type lgssc registered [13953.790315] Lustre: Echo OBD driver; http://www.lustre.org/ [13961.434898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13961.472408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13962.700875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13962.730378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13962.806628] Lustre: lustre-MDT0000: new disk, initializing [13962.882724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13962.900358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13966.420620] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [13975.346221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13975.471706] Lustre: 473313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13975.512115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13975.515257] Lustre: Skipped 1 previous similar message [13975.576311] Lustre: lustre-MDT0001: new disk, initializing [13975.689889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13975.715383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13975.725050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13979.200897] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [13988.069683] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13993.507860] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14001.067858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14001.362040] Lustre: lustre-OST0000: new disk, initializing [14001.370958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14001.378924] Lustre: 475282:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14001.467827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14006.786906] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14010.916602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14010.929491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14011.016276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14015.684837] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14019.346898] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing lnet_if_list [14024.041126] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14024.081039] LNet: Added LNI 192.168.206.133@tcp10 [8/256/0/180] [14030.006817] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14030.044829] LNet: Added LNI 192.168.206.133@tcp11 [8/256/0/180] [14034.616622] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14034.645616] LNet: Added LNI 192.168.206.133@tcp12 [8/256/0/180] [14039.537460] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14039.573079] LNet: Added LNI 192.168.206.133@tcp13 [8/256/0/180] [14044.607956] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14044.632912] LNet: Added LNI 192.168.206.133@tcp14 [8/256/0/180] [14049.389711] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14049.413618] LNet: Added LNI 192.168.206.133@tcp15 [8/256/0/180] [14055.850160] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14055.904646] LNet: Added LNI 192.168.206.133@tcp16 [8/256/0/180] [14062.496170] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14062.521408] LNet: Added LNI 192.168.206.133@tcp17 [8/256/0/180] [14066.916873] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14066.943670] LNet: Added LNI 192.168.206.133@tcp18 [8/256/0/180] [14071.943824] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14071.963309] LNet: Added LNI 192.168.206.133@tcp19 [8/256/0/180] [14076.867780] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14076.903450] LNet: Added LNI 192.168.206.133@tcp20 [8/256/0/180] [14081.947302] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14081.977518] LNet: Added LNI 192.168.206.133@tcp21 [8/256/0/180] [14087.253694] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14087.294347] LNet: Added LNI 192.168.206.133@tcp22 [8/256/0/180] [14092.312247] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14092.337632] LNet: Added LNI 192.168.206.133@tcp23 [8/256/0/180] [14097.405437] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14097.428301] LNet: Added LNI 192.168.206.133@tcp24 [8/256/0/180] [14102.512509] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14102.537474] LNet: Added LNI 192.168.206.133@tcp25 [8/256/0/180] [14108.027249] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14108.058315] LNet: Added LNI 192.168.206.133@tcp26 [8/256/0/180] [14113.221403] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14113.243597] LNet: Added LNI 192.168.206.133@tcp27 [8/256/0/180] [14118.835314] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14118.860442] LNet: Added LNI 192.168.206.133@tcp28 [8/256/0/180] [14124.613742] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14124.645834] LNet: Added LNI 192.168.206.133@tcp29 [8/256/0/180] [14129.763227] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14129.785107] LNet: Added LNI 192.168.206.133@tcp30 [8/256/0/180] [14134.936392] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14134.973656] LNet: Added LNI 192.168.206.133@tcp31 [8/256/0/180] [14140.618987] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14140.648716] LNet: Added LNI 192.168.206.133@tcp32 [8/256/0/180] [14147.436371] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14147.477822] LNet: Added LNI 192.168.206.133@tcp33 [8/256/0/180] [14155.812604] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14155.848513] LNet: Added LNI 192.168.206.133@tcp34 [8/256/0/180] [14163.379176] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14163.427732] LNet: Added LNI 192.168.206.133@tcp35 [8/256/0/180] [14170.380862] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14170.437436] LNet: Added LNI 192.168.206.133@tcp36 [8/256/0/180] [14177.396399] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14177.433468] LNet: Added LNI 192.168.206.133@tcp37 [8/256/0/180] [14183.651053] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14183.690894] LNet: Added LNI 192.168.206.133@tcp38 [8/256/0/180] [14190.453634] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14190.483092] LNet: Added LNI 192.168.206.133@tcp39 [8/256/0/180] [14197.092495] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14197.116746] LNet: Added LNI 192.168.206.133@tcp40 [8/256/0/180] [14203.904661] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14203.924268] LNet: Added LNI 192.168.206.133@tcp41 [8/256/0/180] [14211.170275] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14211.198770] LNet: Added LNI 192.168.206.133@tcp42 [8/256/0/180] [14218.391237] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14218.416864] LNet: Added LNI 192.168.206.133@tcp43 [8/256/0/180] [14225.750132] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14225.770924] LNet: Added LNI 192.168.206.133@tcp44 [8/256/0/180] [14235.250911] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14235.278641] LNet: Added LNI 192.168.206.133@tcp45 [8/256/0/180] [14245.079443] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14245.117452] LNet: Added LNI 192.168.206.133@tcp46 [8/256/0/180] [14254.337425] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14254.372757] LNet: Added LNI 192.168.206.133@tcp47 [8/256/0/180] [14262.729633] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14262.762405] LNet: Added LNI 192.168.206.133@tcp48 [8/256/0/180] [14271.099649] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14271.122681] LNet: Added LNI 192.168.206.133@tcp49 [8/256/0/180] [14278.673093] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14278.729934] LNet: Added LNI 192.168.206.133@tcp50 [8/256/0/180] [14297.570231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14297.577732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14297.593068] Lustre: Skipped 2 previous similar messages [14297.601660] Lustre: Skipped 2 previous similar messages [14301.504444] Lustre: server umount lustre-MDT0000 complete [14304.681911] LustreError: 473308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840458 with bad export cookie 13479183293762518123 [14304.683168] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14304.689966] LustreError: 473308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14305.001170] Lustre: server umount lustre-MDT0001 complete [14323.168108] Lustre: 472587:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840461/real 1787840461] req@00000000d7191480 x1874686226103552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14323.190268] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14323.289631] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [14325.985506] Lustre: 472588:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840463/real 1787840463] req@0000000041e35643 x1874686226103680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14328.288148] Lustre: 472589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840466/real 1787840466] req@00000000930c1958 x1874686226103808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840482 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14330.406238] Lustre: 472589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840468/real 1787840468] req@00000000f8364128 x1874686226104064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840484 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14331.039827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14335.456268] Lustre: 472587:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840473/real 1787840473] req@000000006ab35d21 x1874686226104192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787840489 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14349.860929] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbb0fadca37184697 [14349.892956] Lustre: MGC192.168.206.133@tcp: Connection restored to 0@lo (at 0@lo) [14350.236536] LustreError: 475275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14350.251801] LustreError: 475275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14350.317688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14352.353505] LustreError: 475276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14352.359094] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14353.380705] LustreError: 475275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14355.466562] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14358.515394] LustreError: 475276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14363.645104] LustreError: 475275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14365.347650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14365.832119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14368.951801] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14369.997874] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14377.605776] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14382.587898] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14405.099932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14405.110833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14405.127062] Lustre: Skipped 1 previous similar message [14405.151918] Lustre: Skipped 1 previous similar message [14409.746594] Lustre: server umount lustre-OST0000 complete [14415.334736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14415.372053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14415.376864] Lustre: Skipped 1 previous similar message [14419.885241] Lustre: server umount lustre-MDT0000 complete [14420.454074] LustreError: 500259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14420.490766] LustreError: 500259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14423.980293] LustreError: 473308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840577 with bad export cookie 13479183293762520727 [14423.999436] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14424.001921] LustreError: 473308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14424.406678] Lustre: server umount lustre-MDT0001 complete [14434.139817] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [14436.757775] Key type lgssc unregistered [14437.118408] LNet: 502330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14437.129477] LNetError: 502330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14437.137465] LNet: Removed LNI 192.168.206.133@tcp [14437.140325] LNet: Removed LNI 192.168.206.133@tcp10 [14437.145118] LNet: Removed LNI 192.168.206.133@tcp11 [14437.147936] LNet: Removed LNI 192.168.206.133@tcp12 [14437.160326] LNet: Removed LNI 192.168.206.133@tcp13 [14437.163367] LNet: Removed LNI 192.168.206.133@tcp14 [14437.165775] LNet: Removed LNI 192.168.206.133@tcp15 [14437.168139] LNet: Removed LNI 192.168.206.133@tcp16 [14437.170203] LNet: Removed LNI 192.168.206.133@tcp17 [14437.172486] LNet: Removed LNI 192.168.206.133@tcp18 [14437.174949] LNet: Removed LNI 192.168.206.133@tcp19 [14437.177355] LNet: Removed LNI 192.168.206.133@tcp20 [14437.179757] LNet: Removed LNI 192.168.206.133@tcp21 [14437.182043] LNet: Removed LNI 192.168.206.133@tcp22 [14437.184519] LNet: Removed LNI 192.168.206.133@tcp23 [14437.186648] LNet: Removed LNI 192.168.206.133@tcp24 [14437.192536] LNet: Removed LNI 192.168.206.133@tcp25 [14437.194532] LNet: Removed LNI 192.168.206.133@tcp26 [14437.197256] LNet: Removed LNI 192.168.206.133@tcp27 [14437.199142] LNet: Removed LNI 192.168.206.133@tcp28 [14437.201661] LNet: Removed LNI 192.168.206.133@tcp29 [14437.203946] LNet: Removed LNI 192.168.206.133@tcp30 [14437.206253] LNet: Removed LNI 192.168.206.133@tcp31 [14437.208722] LNet: Removed LNI 192.168.206.133@tcp32 [14437.211325] LNet: Removed LNI 192.168.206.133@tcp33 [14437.213721] LNet: Removed LNI 192.168.206.133@tcp34 [14437.216514] LNet: Removed LNI 192.168.206.133@tcp35 [14437.219360] LNet: Removed LNI 192.168.206.133@tcp36 [14437.222091] LNet: Removed LNI 192.168.206.133@tcp37 [14437.224244] LNet: Removed LNI 192.168.206.133@tcp38 [14437.226387] LNet: Removed LNI 192.168.206.133@tcp39 [14437.228704] LNet: Removed LNI 192.168.206.133@tcp40 [14437.231182] LNet: Removed LNI 192.168.206.133@tcp41 [14437.233620] LNet: Removed LNI 192.168.206.133@tcp42 [14437.235927] LNet: Removed LNI 192.168.206.133@tcp43 [14437.238345] LNet: Removed LNI 192.168.206.133@tcp44 [14437.240906] LNet: Removed LNI 192.168.206.133@tcp45 [14437.243219] LNet: Removed LNI 192.168.206.133@tcp46 [14437.245994] LNet: Removed LNI 192.168.206.133@tcp47 [14437.248513] LNet: Removed LNI 192.168.206.133@tcp48 [14437.250907] LNet: Removed LNI 192.168.206.133@tcp49 [14437.258940] LNet: Removed LNI 192.168.206.133@tcp50 [14438.051232] Key type .llcrypt unregistered [14438.053792] Key type ._llcrypt unregistered [14456.210527] Key type ._llcrypt registered [14456.212935] Key type .llcrypt registered [14456.310582] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [14469.621439] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [14470.629338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14470.720771] alg: No test for adler32 (adler32-zlib) [14471.853455] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [14472.094084] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [14473.760198] Key type lgssc registered [14474.643819] Lustre: Echo OBD driver; http://www.lustre.org/ [14503.080146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14503.099549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14504.246815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14504.277756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14504.334894] Lustre: lustre-MDT0000: new disk, initializing [14504.403160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14504.416990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14510.838665] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14519.455684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14519.550961] Lustre: 505521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14519.604958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14519.608075] Lustre: Skipped 1 previous similar message [14519.669788] Lustre: lustre-MDT0001: new disk, initializing [14519.754615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14519.779933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14519.789784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14523.379691] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14530.845850] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14535.374088] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14541.519271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14541.674720] Lustre: *** cfs_fail_loc=90f, val=0*** [14541.732798] Lustre: lustre-OST0000: new disk, initializing [14541.737872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14541.745683] Lustre: 507533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14541.838503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14546.350699] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14549.559129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14549.570659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14549.690544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14554.053143] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14560.226737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14560.228102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14560.240844] LustreError: Skipped 1 previous similar message [14560.244295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14560.247285] Lustre: Skipped 1 previous similar message [14565.349253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14565.370615] Lustre: Skipped 1 previous similar message [14565.835890] Lustre: server umount lustre-OST0000 complete [14570.466423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14570.489120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14570.496146] Lustre: Skipped 2 previous similar messages [14575.587504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14575.603745] Lustre: Skipped 1 previous similar message [14575.847495] Lustre: server umount lustre-MDT0000 complete [14580.094127] LustreError: 505511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840733 with bad export cookie 15511597438958000932 [14580.095644] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14580.104706] LustreError: 505511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14581.185146] Lustre: server umount lustre-MDT0001 complete [14595.832293] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [14599.676377] Key type lgssc unregistered [14599.878765] LNet: 509441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14599.882700] LNetError: 509441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14600.936097] LNet: Removed LNI 192.168.206.133@tcp [14602.035749] Key type .llcrypt unregistered [14602.044711] Key type ._llcrypt unregistered [14620.663716] Key type ._llcrypt registered [14620.666598] Key type .llcrypt registered [14620.777897] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [14621.730858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14621.901835] alg: No test for adler32 (adler32-zlib) [14622.857331] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [14623.062795] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [14624.688352] Key type lgssc registered [14625.879411] Lustre: Echo OBD driver; http://www.lustre.org/ [14633.030437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14633.043323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14634.371180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14638.783479] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14644.707753] LustreError: 510714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14644.728272] LustreError: 510714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14647.114698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14647.382282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14651.328513] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14657.221794] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14663.808605] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14669.455571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14669.667224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14674.426338] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14680.075795] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14689.411577] loop0: detected capacity change from 0 to 204800000 [14689.617784] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14689.626966] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14689.638185] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14695.619214] loop0: detected capacity change from 0 to 204800000 [14695.867414] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14695.887419] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14695.906423] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14701.004524] loop0: detected capacity change from 0 to 204800000 [14705.916474] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14706.000150] Lustre: fs15246-OST0000: new disk, initializing [14706.006796] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14706.012246] Lustre: 514309:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14706.067578] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14712.330246] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14721.160373] loop1: detected capacity change from 0 to 204800000 [14725.949590] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14726.016519] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14726.058690] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14726.117317] Lustre: fs15246-MDT0000: new disk, initializing [14726.258292] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14726.284933] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14726.370537] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14727.398230] Lustre: 515431:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004892f8da x1874686929578240/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787840897 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14729.937573] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14736.495279] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14736.525288] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14736.552165] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14749.164286] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14749.171213] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14753.500212] Lustre: server umount fs15246-MDT0000 complete [14754.278574] LustreError: 510711:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14759.399297] LustreError: 510711:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14762.365336] Lustre: server umount fs15246-OST0000 complete [14774.763490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14774.764825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14774.778767] Lustre: Skipped 1 previous similar message [14774.796584] Lustre: Skipped 1 previous similar message [14778.581050] Lustre: server umount lustre-OST0000 complete [14784.997652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14785.000525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14785.018476] Lustre: Skipped 1 previous similar message [14785.034879] Lustre: Skipped 1 previous similar message [14788.458897] Lustre: server umount lustre-MDT0000 complete [14790.116244] LustreError: 510715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14790.132897] LustreError: 510715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14791.743890] LustreError: 510694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840945 with bad export cookie 7186682053716208344 [14791.751438] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14791.754163] LustreError: 510694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14792.159552] Lustre: server umount lustre-MDT0001 complete [14801.250583] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [14803.795493] Key type lgssc unregistered [14804.151663] LNet: 518528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14804.162414] LNetError: 518528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14804.190833] LNet: Removed LNI 192.168.206.133@tcp [14804.782234] Key type .llcrypt unregistered [14804.784089] Key type ._llcrypt unregistered [14822.922262] Key type ._llcrypt registered [14822.925792] Key type .llcrypt registered [14823.056282] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [14824.393180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14824.511127] alg: No test for adler32 (adler32-zlib) [14825.558360] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [14825.865954] LNet: Added LNI 192.168.206.133@tcp [8/256/0/180] [14827.664302] Key type lgssc registered [14829.514432] Lustre: Echo OBD driver; http://www.lustre.org/ [14864.681246] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [14873.399562] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [14906.323098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14906.376366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14907.755035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14907.825735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14907.965919] Lustre: lustre-MDT0000: new disk, initializing [14908.097381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14908.123386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14912.725665] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14923.010525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14923.154797] Lustre: 522990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14923.240282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14923.256608] Lustre: Skipped 1 previous similar message [14923.355738] Lustre: lustre-MDT0001: new disk, initializing [14923.469397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14923.532095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14923.541863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14927.804719] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14937.636886] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14944.467903] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14954.168758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14954.466757] Lustre: lustre-OST0000: new disk, initializing [14954.470746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14954.479301] Lustre: 524963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14954.561278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14963.108883] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_default_debug -1 all [14964.321819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14964.327873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14964.410342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14972.105748] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14976.266906] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14976.449640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14980.561403] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14980.849261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14985.186340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14985.189978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14985.194784] LustreError: Skipped 1 previous similar message [14985.196257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14985.209414] Lustre: Skipped 1 previous similar message [14985.222820] Lustre: Skipped 1 previous similar message [14989.569415] Lustre: server umount lustre-OST0000 complete [14995.428714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14995.431156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14995.447804] Lustre: Skipped 1 previous similar message [14999.398544] Lustre: server umount lustre-MDT0000 complete [15002.565386] LustreError: 523787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841156 with bad export cookie 4591892612113922341 [15002.570048] LustreError: MGC192.168.206.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15002.574235] LustreError: 523787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15002.896825] Lustre: server umount lustre-MDT0001 complete [15025.756837] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing set_hostid [15035.439767] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing load_modules_local [15067.782242] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14801 sec ========= 10:33:39 (1787841219) [15069.255400] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:33:41 (1787841221) === [15072.790386] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:33:44 (1787841224) === [15120.322755] Lustre: DEBUG MARKER: oleg633-server.virtnet: executing unload_modules_local [15123.519372] Key type lgssc unregistered [15123.854936] LNet: 531613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15123.864742] LNetError: 531613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15123.893968] LNet: Removed LNI 192.168.206.133@tcp [15124.726114] Key type .llcrypt unregistered [15124.729989] Key type ._llcrypt unregistered