[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469012470 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002013] Switched APIC routing to physical x2apic. [ 0.003015] kvm-guest: setup PV IPIs [ 0.005751] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007016] pid_max: default: 32768 minimum: 301 [ 0.009061] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012053] *** VALIDATE selinux *** [ 0.018809] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022692] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025120] *** VALIDATE tmpfs *** [ 0.027336] *** VALIDATE proc *** [ 0.028183] *** VALIDATE cgroup *** [ 0.029007] *** VALIDATE cgroup2 *** [ 0.030242] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032040] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034034] Spectre V2 : User space: Vulnerable [ 0.035010] Speculative Store Bypass: Vulnerable [ 0.038313] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.040204] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042027] ... version: 2 [ 0.043016] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045015] ... value mask: 0000ffffffffffff [ 0.046013] ... max period: 00007fffffffffff [ 0.047012] ... fixed-purpose events: 3 [ 0.048012] ... event mask: 000000070000000f [ 0.049333] rcu: Hierarchical SRCU implementation. [ 0.051456] smp: Bringing up secondary CPUs ... [ 0.052423] x86: Booting SMP configuration: [ 0.053018] .... node #0, CPUs: #1 #2 #3 [ 0.056355] smp: Brought up 1 node, 4 CPUs [ 0.058017] smpboot: Max logical packages: 1 [ 0.059015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150020] node 0 deferred pages initialised in 87ms [ 0.153453] devtmpfs: initialized [ 0.154216] x86/mm: Memory block size: 128MB [ 0.156000] gcov: version magic: 0x41383552 [ 0.159289] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162567] pinctrl core: initialized pinctrl subsystem [ 0.165223] [ 0.165779] ************************************************************* [ 0.168015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172015] ** ** [ 0.174013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177015] ** ** [ 0.180015] ** This means that this kernel is built to expose internal ** [ 0.182012] ** IOMMU data structures, which may compromise security on ** [ 0.185014] ** your system. ** [ 0.188014] ** ** [ 0.190025] ** If you see this message and you are not debugging the ** [ 0.193015] ** kernel, report this immediately to your vendor! ** [ 0.196012] ** ** [ 0.198012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200011] ************************************************************* [ 0.204000] NET: Registered protocol family 16 [ 0.206584] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210113] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.214096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.219044] cpuidle: using governor menu [ 0.220561] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.224420] PCI: Using configuration type 1 for base access [ 0.226127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243159] cryptd: max_cpu_qlen set to 1000 [ 0.247337] ACPI: Added _OSI(Module Device) [ 0.249032] ACPI: Added _OSI(Processor Device) [ 0.251024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.254028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.259194] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.268560] ACPI: Interpreter enabled [ 0.270097] ACPI: PM: (supports S0 S3 S4 S5) [ 0.272024] ACPI: Using IOAPIC for interrupt routing [ 0.273164] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.277362] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.287986] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.291081] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.298121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.304723] acpiphp: Slot [2] registered [ 0.306246] acpiphp: Slot [5] registered [ 0.308123] acpiphp: Slot [6] registered [ 0.310202] acpiphp: Slot [7] registered [ 0.311182] acpiphp: Slot [8] registered [ 0.313138] acpiphp: Slot [9] registered [ 0.315248] acpiphp: Slot [10] registered [ 0.317209] acpiphp: Slot [3] registered [ 0.318134] acpiphp: Slot [4] registered [ 0.320208] acpiphp: Slot [11] registered [ 0.322155] acpiphp: Slot [12] registered [ 0.323141] acpiphp: Slot [13] registered [ 0.325181] acpiphp: Slot [14] registered [ 0.327204] acpiphp: Slot [15] registered [ 0.328142] acpiphp: Slot [16] registered [ 0.330142] acpiphp: Slot [17] registered [ 0.331094] acpiphp: Slot [18] registered [ 0.333135] acpiphp: Slot [19] registered [ 0.334096] acpiphp: Slot [20] registered [ 0.336167] acpiphp: Slot [21] registered [ 0.337140] acpiphp: Slot [22] registered [ 0.339124] acpiphp: Slot [23] registered [ 0.340101] acpiphp: Slot [24] registered [ 0.342175] acpiphp: Slot [25] registered [ 0.344144] acpiphp: Slot [26] registered [ 0.346156] acpiphp: Slot [27] registered [ 0.348163] acpiphp: Slot [28] registered [ 0.349186] acpiphp: Slot [29] registered [ 0.351169] acpiphp: Slot [30] registered [ 0.353167] acpiphp: Slot [31] registered [ 0.355085] PCI host bridge to bus 0000:00 [ 0.357096] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.360031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.362039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.364030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.367035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.369038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.371219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.374631] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.378792] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.390737] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.396050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.398025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.400025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.401000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.403670] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.408110] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.413242] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.421024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.436029] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.442026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.449292] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.456024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.464024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.493421] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.512019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.529022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.541842] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.550021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.556023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.575020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.585218] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.592024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.597026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.635074] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.642031] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.650029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.673024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.685191] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.709021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.722024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.750027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.763433] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.769611] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.771431] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.774462] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.776349] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.781176] iommu: Default domain type: Passthrough [ 0.783474] SCSI subsystem initialized [ 0.786231] ACPI: bus type USB registered [ 0.787200] usbcore: registered new interface driver usbfs [ 0.789203] usbcore: registered new interface driver hub [ 0.791120] usbcore: registered new device driver usb [ 0.793182] pps_core: LinuxPPS API ver. 1 registered [ 0.794013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.797086] PTP clock support registered [ 0.799169] EDAC MC: Ver: 3.0.0 [ 0.801570] PCI: Using ACPI for IRQ routing [ 0.805012] NetLabel: Initializing [ 0.806015] NetLabel: domain hash size = 128 [ 0.808015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.810109] NetLabel: unlabeled traffic allowed by default [ 0.813199] vgaarb: loaded [ 0.815328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.817020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.824000] clocksource: Switched to clocksource kvm-clock [ 0.936357] VFS: Disk quotas dquot_6.6.0 [ 0.938197] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.941627] *** VALIDATE ramfs *** [ 0.942944] *** VALIDATE hugetlbfs *** [ 0.944771] pnp: PnP ACPI init [ 0.947355] pnp: PnP ACPI: found 6 devices [ 0.967592] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.971143] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.973171] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.975668] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.977811] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.980022] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.983173] NET: Registered protocol family 2 [ 0.985695] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.990495] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.995790] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.000894] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.004306] TCP: Hash tables configured (established 65536 bind 65536) [ 1.007395] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.010795] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.013599] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016864] NET: Registered protocol family 1 [ 1.019856] RPC: Registered named UNIX socket transport module. [ 1.022134] RPC: Registered udp transport module. [ 1.023846] RPC: Registered tcp transport module. [ 1.025560] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.027995] NET: Registered protocol family 44 [ 1.029759] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.032078] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.034169] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.036625] PCI: CLS 0 bytes, default 64 [ 1.038679] Unpacking initramfs... [ 2.472457] debug: unmapping init [mem 0xffff95277cc54000-0xffff95277ffbffff] [ 2.480209] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.482923] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.486897] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.008185] Initialise system trusted keyrings [ 3.010233] Key type blacklist registered [ 3.017111] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.036580] zbud: loaded [ 3.042128] *** VALIDATE nfs *** [ 3.043838] *** VALIDATE nfs4 *** [ 3.045632] pstore: using deflate compression [ 3.050349] Platform Keyring initialized [ 3.172500] NET: Registered protocol family 38 [ 3.176377] Key type asymmetric registered [ 3.178262] Asymmetric key parser 'x509' registered [ 3.181092] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.183981] io scheduler mq-deadline registered [ 3.185968] io scheduler kyber registered [ 3.187876] io scheduler bfq registered [ 3.190179] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.193561] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.196779] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.199824] ACPI: Power Button [PWRF] [ 3.206941] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.214738] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.232961] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.240804] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.261137] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.289873] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.323253] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.328982] Non-volatile memory driver v1.3 [ 3.331132] Linux agpgart interface v0.103 [ 3.369215] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.372907] vda: detected capacity change from 0 to 74645504 [ 3.390194] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.394170] vdb: detected capacity change from 0 to 1073741824 [ 3.411905] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.415962] vdc: detected capacity change from 0 to 2621440000 [ 3.434534] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.437673] vdd: detected capacity change from 0 to 2621440000 [ 3.453707] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.457011] vde: detected capacity change from 0 to 4294967296 [ 3.473351] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.476723] vdf: detected capacity change from 0 to 4294967296 [ 3.485792] libphy: Fixed MDIO Bus: probed [ 3.491550] usbcore: registered new interface driver usbserial_generic [ 3.494449] usbserial: USB Serial support registered for generic [ 3.497434] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.502548] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.504443] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.507649] mousedev: PS/2 mouse device common for all mice [ 3.511482] rtc_cmos 00:05: RTC can wake from S4 [ 3.511575] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.515446] rtc_cmos 00:05: registered as rtc0 [ 3.521167] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.521685] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.527115] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.528746] intel_pstate: CPU model not supported [ 3.536729] hid: raw HID events driver (C) Jiri Kosina [ 3.539192] usbcore: registered new interface driver usbhid [ 3.541588] usbhid: USB HID core driver [ 3.543637] drop_monitor: Initializing network drop monitor service [ 3.546489] Initializing XFRM netlink socket [ 3.550442] NET: Registered protocol family 10 [ 3.554160] Segment Routing with IPv6 [ 3.555883] NET: Registered protocol family 17 [ 3.558493] mpls_gso: MPLS GSO support [ 3.567258] RAS: Correctable Errors collector initialized. [ 3.570733] AVX version of gcm_enc/dec engaged. [ 3.572948] AES CTR mode by8 optimization enabled [ 3.656613] sched_clock: Marking stable (3656570999, 0)->(4643698556, -987127557) [ 3.660746] registered taskstats version 1 [ 3.663137] Loading compiled-in X.509 certificates [ 3.665569] zswap: loaded using pool lzo/zbud [ 3.691970] Key type big_key registered [ 3.707963] Key type encrypted registered [ 3.709825] ima: No TPM chip found, activating TPM-bypass! [ 3.712352] ima: Allocated hash algorithm: sha1 [ 3.714363] ima: No architecture policies found [ 3.716377] evm: Initialising EVM extended attributes: [ 3.718418] evm: security.selinux [ 3.719835] evm: security.ima [ 3.720946] evm: security.capability [ 3.722467] evm: HMAC attrs: 0x1 [ 3.735058] rtc_cmos 00:05: setting system clock to 2026-07-16 06:54:51 UTC (1784184891) [ 3.744681] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.748417] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.761146] Write protecting the kernel read-only data: 28672k [ 3.764936] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.768239] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.802960] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.812901] systemd[1]: Detected virtualization kvm. [ 3.815634] systemd[1]: Detected architecture x86-64. [ 3.818401] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.844896] systemd[1]: No hostname configured. [ 3.846417] systemd[1]: Set hostname to . [ 3.848604] random: systemd: uninitialized urandom read (16 bytes read) [ 3.850504] systemd[1]: Initializing machine ID from random generator. [ 3.904701] random: ln: uninitialized urandom read (6 bytes read) [ 4.007254] random: systemd: uninitialized urandom read (16 bytes read) [ 4.010389] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.015640] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.020265] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.700126] device-mapper: uevent: version 1.0.3 [ 4.702529] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 5.047509] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.446220] virtio_net virtio0 ens2: renamed from eth0 [ 5.541201] scsi host0: ata_piix [ 5.578392] scsi host1: ata_piix [ 5.580456] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.583406] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.248688] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.240665] random: crng init done [ 10.243251] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.787869] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.063787] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.360919] SELinux: Disabled at runtime. [ 12.428076] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.440846] systemd[1]: Detected virtualization kvm. [ 12.443207] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.973395] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.977380] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.987473] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.991708] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.996874] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.005299] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.022510] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ 13.172933] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.524683] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.922255] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.017322] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.125656] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.143384] EDAC sbridge: Ver: 1.1.2 [ 15.877480] Key type dns_resolver registered [ 16.187665] NFS: Registering the id_resolver key type [ 16.189810] Key type id_resolver registered [ 16.191676] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg420-server login: [ 32.988257] hrtimer: interrupt took 4310964 ns [ 78.569900] libcfs: loading out-of-tree module taints kernel. [ 78.612883] Key type ._llcrypt registered [ 78.614662] Key type .llcrypt registered [ 78.685908] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 100.274251] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 102.397119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.422684] alg: No test for adler32 (adler32-zlib) [ 103.970469] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 104.972381] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 106.871484] Key type lgssc registered [ 109.233465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.553329] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.947809] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 180.893350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.964304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.331671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.398771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.552280] Lustre: lustre-MDT0000: new disk, initializing [ 182.633327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.677511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.680701] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.597975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.687835] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.712955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.717254] Lustre: Skipped 1 previous similar message [ 202.811488] Lustre: lustre-MDT0001: new disk, initializing [ 202.899764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.927991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.936208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.025852] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.668163] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.877683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.249381] Lustre: lustre-OST0000: new disk, initializing [ 225.253421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.260598] Lustre: 8451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.352989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.910966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 228.923092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.034680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.766366] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.643582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.831687] Lustre: lustre-OST0001: new disk, initializing [ 249.842111] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.857755] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.966507] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.044701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 257.068060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 257.190860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 257.333322] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.904257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 278.944870] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.587915] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.472470] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing yml_node [ 298.401129] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 301.998249] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 305.172615] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 307.283133] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 16 02:59:52 EDT 2026 [ 328.818052] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 330.515226] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 349.156680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.158099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.172249] Lustre: Skipped 1 previous similar message [ 349.185313] Lustre: Skipped 3 previous similar messages [ 353.806090] Lustre: server umount lustre-MDT0000 complete [ 358.122606] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185245 with bad export cookie 14317461854636980237 [ 358.128423] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.147338] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 359.393929] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.395873] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.414410] Lustre: Skipped 2 previous similar messages [ 359.415405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.443414] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 359.467684] Lustre: Skipped 1 previous similar message [ 363.433561] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.436717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.466334] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 363.494166] Lustre: Skipped 1 previous similar message [ 364.737760] Lustre: server umount lustre-MDT0001 complete [ 369.146453] Lustre: server umount lustre-OST0000 complete [ 373.220587] Lustre: server umount lustre-OST0001 complete [ 380.995585] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 391.512207] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 434.350531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.844946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 434.888601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 435.008279] Lustre: lustre-MDT0000: new disk, initializing [ 435.090476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.108088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 441.181634] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 453.521786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.711191] Lustre: 15097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 453.830717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 453.837035] Lustre: Skipped 1 previous similar message [ 454.005289] Lustre: lustre-MDT0001: new disk, initializing [ 454.156051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.237559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 454.255034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.698943] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 471.898594] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 478.817401] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 487.936986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.529663] Lustre: lustre-OST0000: new disk, initializing [ 488.539505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.554302] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.711097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.435627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 490.451615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.649282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 496.273646] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 506.583715] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 512.291963] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 512.526657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 516.839673] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 517.222207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 521.189399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.197878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.215914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.247482] Lustre: Skipped 2 previous similar messages [ 521.265927] Lustre: Skipped 1 previous similar message [ 526.277304] Lustre: server umount lustre-OST0000 complete [ 536.042796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.067282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.100088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.271187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 546.574926] Lustre: server umount lustre-MDT0000 complete [ 547.811788] LustreError: 16359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.847425] LustreError: 16359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 551.063922] LustreError: 15892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185438 with bad export cookie 14317461854636981777 [ 551.067199] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.078443] LustreError: 15892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 551.672544] Lustre: server umount lustre-MDT0001 complete [ 556.849505] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:04:01 (1784185441) [ 559.482941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 561.787171] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:04:06 (1784185446) [ 569.301308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.978602] LustreError: 19477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.078210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.457330] LustreError: 19478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.498398] LustreError: 19478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 575.608491] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 584.673949] LustreError: 19477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.709143] LustreError: 19477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 585.288592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.398390] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 602.551096] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 610.065756] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 618.497306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 626.504890] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 636.068712] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 651.744763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 651.757907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.771117] Lustre: Skipped 1 previous similar message [ 651.781953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 651.799983] Lustre: Skipped 5 previous similar messages [ 654.824041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 658.039356] Lustre: server umount lustre-OST0000 complete [ 662.505877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 662.518318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.550403] Lustre: Skipped 1 previous similar message [ 669.036370] Lustre: server umount lustre-MDT0000 complete [ 670.176389] LustreError: 20418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.210543] LustreError: 20418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 673.460402] LustreError: 21322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185561 with bad export cookie 14317461854636982659 [ 673.460882] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.477650] LustreError: 21322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 673.817358] Lustre: server umount lustre-MDT0001 complete [ 691.538792] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 694.837791] Key type lgssc unregistered [ 695.231632] LNet: 23061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.242828] LNetError: 23061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.281139] LNet: Removed LNI 192.168.204.120@tcp [ 696.525145] Key type .llcrypt unregistered [ 696.526912] Key type ._llcrypt unregistered [ 726.926832] Key type ._llcrypt registered [ 726.928638] Key type .llcrypt registered [ 727.031412] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 728.230097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 728.287319] alg: No test for adler32 (adler32-zlib) [ 729.453194] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 729.877945] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 731.663678] Key type lgssc registered [ 732.685408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 741.401190] loop: module loaded [ 741.488489] loop0: detected capacity change from 0 to 204800000 [ 741.712096] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 741.731974] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 742.201227] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.208892] loop0: detected capacity change from 0 to 204800000 [ 751.502462] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.530300] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 751.554718] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 758.772633] loop0: detected capacity change from 0 to 204800000 [ 765.870616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 765.904708] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 767.286887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 767.366975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 767.495860] Lustre: lustre-MDT0000: new disk, initializing [ 767.585508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 767.615526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 773.088896] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 784.207494] loop1: detected capacity change from 0 to 204800000 [ 790.676163] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 791.011394] Lustre: lustre-OST07c6: new disk, initializing [ 791.015805] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 791.020773] Lustre: Skipped 1 previous similar message [ 791.025528] Lustre: 26249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 791.140975] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 798.898067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 798.914491] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 798.983181] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 799.218319] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 805.040078] Lustre: 25238:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 805.070105] Lustre: 25238:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 805.091101] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 805.099426] Lustre: 25238:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 805.110663] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 805.115748] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 823.895033] Lustre: 25237:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 823.911111] Lustre: 25237:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 823.928261] Lustre: 25237:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 823.942504] Lustre: 25237:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 823.950768] Lustre: 25237:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 823.962413] Lustre: 25237:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 829.084507] Lustre: 25238:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 829.105283] Lustre: 25238:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 829.114990] Lustre: 25238:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 829.121906] Lustre: 25238:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 829.129732] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 829.137235] Lustre: 25238:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 829.147331] Lustre: 25238:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 829.156797] Lustre: 25238:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 829.161208] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 829.167712] Lustre: 25238:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 829.172633] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 829.178091] Lustre: 25238:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.397633] Lustre: 25237:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 832.414401] Lustre: 25237:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 832.429376] Lustre: 25237:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 832.440750] Lustre: 25237:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.460589] Lustre: 25237:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 832.470994] Lustre: 25237:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.488183] Lustre: 25237:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 832.498866] Lustre: 25237:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.507652] Lustre: 25237:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 832.522438] Lustre: 25237:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.532313] Lustre: 25237:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 832.541353] Lustre: 25237:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 837.701211] Lustre: 25237:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 837.712499] Lustre: 25237:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 837.721319] Lustre: 25237:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 837.727405] Lustre: 25237:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 837.736547] Lustre: 25237:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 837.747533] Lustre: 25237:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 837.755221] Lustre: 25237:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 837.765378] Lustre: 25237:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 837.780523] Lustre: 25237:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 837.786330] Lustre: 25237:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 837.793889] Lustre: 25237:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 837.803685] Lustre: 25237:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 846.488175] Lustre: 26878:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 846.502316] Lustre: 26878:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 846.525154] Lustre: 26878:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 846.537653] Lustre: 26878:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.545104] Lustre: 26878:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 846.550429] Lustre: 26878:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.567135] Lustre: 26878:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 846.583581] Lustre: 26878:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.604332] Lustre: 26878:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 846.623168] Lustre: 26878:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.642855] Lustre: 26878:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 846.658623] Lustre: 26878:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 900.129494] Lustre: server umount lustre-OST07c6 complete [ 921.249267] Lustre: server umount lustre-MDT0000 complete [ 940.163621] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 943.515754] Key type lgssc unregistered [ 943.797772] LNet: 28810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.807440] LNetError: 28810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.818951] LNet: Removed LNI 192.168.204.120@tcp [ 944.802714] Key type .llcrypt unregistered [ 944.805768] Key type ._llcrypt unregistered [ 969.077856] Key type ._llcrypt registered [ 969.081270] Key type .llcrypt registered [ 969.188882] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 970.367488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 970.391676] alg: No test for adler32 (adler32-zlib) [ 971.528900] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 971.788067] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 973.480904] Key type lgssc registered [ 974.527164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 984.299059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 984.346039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.028708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.663277] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 995.304306] LustreError: 30080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.329534] LustreError: 30080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1000.418231] LustreError: 30081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.011989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.571942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1007.464704] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1016.887341] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1024.157081] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1033.303352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.824419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.608143] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1049.570784] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1055.347782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1074.145701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.162135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.194794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.219303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1076.223177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.223791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.245083] Lustre: Skipped 1 previous similar message [ 1079.600420] Lustre: server umount lustre-OST0000 complete [ 1086.438620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.452400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1086.478790] Lustre: Skipped 1 previous similar message [ 1086.501720] Lustre: Skipped 1 previous similar message [ 1091.587366] LustreError: 30081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.642193] LustreError: 30081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1091.773621] Lustre: server umount lustre-MDT0000 complete [ 1096.674895] LustreError: 30081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.699496] LustreError: 30081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1097.955615] LustreError: 30062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784185985 with bad export cookie 9364401797833246086 [ 1097.972206] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1097.974602] LustreError: 30062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1098.296752] Lustre: server umount lustre-MDT0001 complete [ 1113.638393] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1117.096778] Key type lgssc unregistered [ 1117.384424] LNet: 33842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.401477] LNetError: 33842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.438577] LNet: Removed LNI 192.168.204.120@tcp [ 1119.273772] Key type .llcrypt unregistered [ 1119.276697] Key type ._llcrypt unregistered [ 1141.885720] Key type ._llcrypt registered [ 1141.887818] Key type .llcrypt registered [ 1141.993104] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1143.226151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1143.402298] alg: No test for adler32 (adler32-zlib) [ 1144.641063] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 1145.007138] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1146.767188] Key type lgssc registered [ 1148.124983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1157.760147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1157.833849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.355141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1165.128726] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1169.897283] LustreError: 35116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.927146] LustreError: 35116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1174.878796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.009090] LustreError: 35115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.271790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1180.057723] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1188.980200] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1195.696487] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1203.099156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1203.508600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.557890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1210.289328] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1219.178866] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1235.430039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.436574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1235.446823] Lustre: Skipped 1 previous similar message [ 1235.454557] Lustre: Skipped 1 previous similar message [ 1238.544037] Lustre: server umount lustre-OST0000 complete [ 1245.668616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.674540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1245.687814] Lustre: Skipped 1 previous similar message [ 1245.703090] Lustre: Skipped 1 previous similar message [ 1249.381892] Lustre: server umount lustre-MDT0000 complete [ 1250.784404] LustreError: 35115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.810569] LustreError: 35115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1253.692718] LustreError: 35094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186141 with bad export cookie 16865995188756321230 [ 1253.701043] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1253.703223] LustreError: 35094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1254.070664] Lustre: server umount lustre-MDT0001 complete [ 1266.297496] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1269.235483] Key type lgssc unregistered [ 1269.617483] LNet: 38668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.631520] LNetError: 38668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.653087] LNet: Removed LNI 192.168.204.120@tcp [ 1270.670306] Key type .llcrypt unregistered [ 1270.675310] Key type ._llcrypt unregistered [ 1296.938878] Key type ._llcrypt registered [ 1296.940719] Key type .llcrypt registered [ 1297.092077] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1298.307781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1298.498363] alg: No test for adler32 (adler32-zlib) [ 1299.627392] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 1299.930920] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1301.655319] Key type lgssc registered [ 1303.037348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1310.919353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1310.940134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.498101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.604601] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1322.976557] LustreError: 40087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.998523] LustreError: 40087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1327.213588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.660631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1332.921462] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1342.222827] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1349.297770] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1358.343106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1358.914967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.309080] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1375.719918] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1382.453725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1387.497547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.502889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1387.537320] Lustre: Skipped 1 previous similar message [ 1389.718299] Lustre: server umount lustre-MDT0000 complete [ 1392.612564] LustreError: 41239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.631330] LustreError: 41239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1393.606336] LustreError: 40069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186281 with bad export cookie 13729665940336509035 [ 1393.612389] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1393.621302] LustreError: 40069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1397.733261] LustreError: 40082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.736524] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.759300] LustreError: 40082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1397.799123] Lustre: Skipped 1 previous similar message [ 1397.804887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1397.812632] Lustre: Skipped 3 previous similar messages [ 1400.047846] Lustre: server umount lustre-MDT0001 complete [ 1421.883444] Lustre: server umount lustre-OST0000 complete [ 1442.583520] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1445.889823] Key type lgssc unregistered [ 1446.311396] LNet: 43962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.330705] LNetError: 43962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.348809] LNet: Removed LNI 192.168.204.120@tcp [ 1447.391127] Key type .llcrypt unregistered [ 1447.400072] Key type ._llcrypt unregistered [ 1477.158834] Key type ._llcrypt registered [ 1477.163091] Key type .llcrypt registered [ 1477.301343] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1478.736785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1478.764468] alg: No test for adler32 (adler32-zlib) [ 1479.926584] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 1480.257302] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1482.120760] Key type lgssc registered [ 1483.355026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1490.756704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1490.780202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.151381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.888560] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1501.676130] LustreError: 45380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.710593] LustreError: 45380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1506.794569] LustreError: 45381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.618460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.935358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1514.410588] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1522.439412] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1529.821156] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1537.456153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1537.882016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.700390] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1553.658441] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1561.475675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1571.809381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.820951] Lustre: Skipped 1 previous similar message [ 1571.822358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1571.835858] Lustre: Skipped 1 previous similar message [ 1575.866454] Lustre: lustre-OST0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 1576.937329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1577.297378] Lustre: server umount lustre-OST0000 complete [ 1595.872849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1595.884234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.901747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.909187] Lustre: Skipped 1 previous similar message [ 1597.413830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.573441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.586795] Lustre: Skipped 3 previous similar messages [ 1610.207119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1610.384577] Lustre: server umount lustre-MDT0000 complete [ 1611.747472] LustreError: 45381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1611.761908] LustreError: 45381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1613.765979] LustreError: 45362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186501 with bad export cookie 16194327211948965854 [ 1613.776304] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.784031] LustreError: 45362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1628.127227] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1628.368716] Lustre: server umount lustre-MDT0001 complete [ 1639.420471] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1641.912878] Key type lgssc unregistered [ 1642.270454] LNet: 49111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.279174] LNetError: 49111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1642.300198] LNet: Removed LNI 192.168.204.120@tcp [ 1643.386131] Key type .llcrypt unregistered [ 1643.391454] Key type ._llcrypt unregistered [ 1670.090855] Key type ._llcrypt registered [ 1670.093907] Key type .llcrypt registered [ 1670.204778] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1671.133143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1671.158842] alg: No test for adler32 (adler32-zlib) [ 1672.160982] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 1672.410888] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1674.079236] Key type lgssc registered [ 1675.112773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1683.365076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1683.404647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.973748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.647664] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1695.201664] LustreError: 50529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.220585] LustreError: 50529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1698.174104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.529196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1702.974922] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1709.888031] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1716.125105] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1722.427319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1722.671843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.807849] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1729.851108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1738.987612] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1746.509413] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1750.772898] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1763.583058] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-16 3h24m07s [ 1765.211365] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1767.869798] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 1768.929649] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 1770.476965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1770.487346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.504622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1773.982780] LustreError: 53184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1774.023812] Lustre: server umount lustre-MDT0000 complete [ 1777.428588] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1777.669641] LustreError: 50528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1780.128775] LustreError: 50524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1780.149809] LustreError: 50524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1784.918378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.029697] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.408928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.500551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1786.419754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1789.813583] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1793.630614] Lustre: DEBUG MARKER: Wait for df (36767) ... [ 1795.405310] Lustre: DEBUG MARKER: done [ 1800.681957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1800.682869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.700860] LustreError: Skipped 1 previous similar message [ 1800.702333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.702339] Lustre: Skipped 5 previous similar messages [ 1800.743182] Lustre: Skipped 3 previous similar messages [ 1805.553793] Lustre: server umount lustre-OST0000 complete [ 1811.957347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.957700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1811.976853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.976861] Lustre: Skipped 3 previous similar messages [ 1811.990322] Lustre: Skipped 1 previous similar message [ 1815.938919] Lustre: server umount lustre-MDT0000 complete [ 1819.105521] LustreError: 50529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.132832] LustreError: 50529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1821.344105] LustreError: 50510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186709 with bad export cookie 16155065018661267827 [ 1821.348293] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1821.365925] LustreError: 50510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1821.924753] Lustre: server umount lustre-MDT0001 complete [ 1838.372739] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1841.510476] Key type lgssc unregistered [ 1841.820336] LNet: 55461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.833197] LNetError: 55461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.858425] LNet: Removed LNI 192.168.204.120@tcp [ 1842.715291] Key type .llcrypt unregistered [ 1842.717624] Key type ._llcrypt unregistered [ 1883.957492] Key type ._llcrypt registered [ 1883.961631] Key type .llcrypt registered [ 1884.099835] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1885.489461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1885.740746] alg: No test for adler32 (adler32-zlib) [ 1886.968612] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 1887.325708] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1888.999227] Key type lgssc registered [ 1890.483791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1898.908715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1898.946098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.258535] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1900.288399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1900.508507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.445519] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1914.615448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.723695] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1914.762922] Lustre: 57671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1915.103303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1920.161546] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1928.770609] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1934.989778] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1941.641316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1941.927762] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1942.120302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.343697] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1950.347114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1955.429563] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1963.113316] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-16 3h27m27s [ 1965.038299] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1972.791811] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1975.734400] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1976.109199] LustreError: 57676:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1996.260183] Lustre: lustre-MDT0000: Client a72e12ec-15ae-48b3-9fe5-ce8bc7074e73 (at 192.168.204.20@tcp) reconnecting [ 2016.183242] LustreError: 57676:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2019.054422] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2021.972971] Lustre: DEBUG MARKER: done [ 2032.097928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2032.114600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.127840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.611804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2032.629779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.643799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.753422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.768310] Lustre: Skipped 1 previous similar message [ 2042.337582] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2042.574459] Lustre: server umount lustre-OST0000 complete [ 2047.971334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.973367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.984694] Lustre: Skipped 1 previous similar message [ 2048.007784] Lustre: Skipped 1 previous similar message [ 2052.821908] Lustre: server umount lustre-MDT0000 complete [ 2056.585547] LustreError: 60428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186944 with bad export cookie 12689343316720165362 [ 2056.594604] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.601927] LustreError: 60428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2056.943939] Lustre: server umount lustre-MDT0001 complete [ 2068.168706] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2070.856373] Key type lgssc unregistered [ 2071.205767] LNet: 62001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.212278] LNetError: 62001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.235247] LNet: Removed LNI 192.168.204.120@tcp [ 2072.238756] Key type .llcrypt unregistered [ 2072.241681] Key type ._llcrypt unregistered [ 2120.769411] Key type ._llcrypt registered [ 2120.772512] Key type .llcrypt registered [ 2120.882577] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2121.875892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.924934] alg: No test for adler32 (adler32-zlib) [ 2123.200324] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 2123.535710] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2125.207298] Key type lgssc registered [ 2126.444285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2129.814394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2129.846724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.267399] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2131.282825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2131.468393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.470564] Lustre: Failing over lustre-MDT0000 [ 2132.845166] Lustre: server umount lustre-MDT0000 complete [ 2139.585757] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:30:24 (1784187024) [ 2145.476236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.856760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.700652] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2157.249215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.370552] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2157.431724] Lustre: 64800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2157.676164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2161.830746] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2169.652979] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2175.610437] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2182.517172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.792182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2183.008185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.995402] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2191.480601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2196.365826] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2201.598566] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2208.744911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2208.760372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.781268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2212.327203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2212.337987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.375025] Lustre: Skipped 1 previous similar message [ 2214.182300] Lustre: server umount lustre-MDT0000 complete [ 2218.224065] LustreError: 64791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187105 with bad export cookie 851608184434123690 [ 2218.227190] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.238944] LustreError: 64791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2218.775965] Lustre: server umount lustre-MDT0001 complete [ 2222.591149] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2227.398280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2236.045750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.943230] Lustre: 63725:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187110/real 1784187110] req@0000000084ffb0f5 x1870855679287168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2238.973162] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.967370] Lustre: 63722:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187111/real 1784187111] req@00000000e61542c8 x1870855679287296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2244.063153] Lustre: 63723:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187115/real 1784187115] req@000000009cdc1617 x1870855679287424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2244.128310] LustreError: 63721:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003ac05560 x1870855679288320/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2244.523325] LustreError: 66645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.548688] LustreError: 66645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2244.655432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.762770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2248.675628] LustreError: 66646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.678478] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2249.697373] Lustre: 63724:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187121/real 1784187121] req@0000000004c5115a x1870855679287808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2249.704383] LustreError: 68565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.730043] Lustre: 63724:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2249.766534] LustreError: 68565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2254.815174] Lustre: 63723:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187126/real 1784187126] req@0000000001f93d56 x1870855679287936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187142 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2254.817258] LustreError: 68567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.863615] LustreError: 68567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2258.637141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.069903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2263.603106] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2264.564605] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2267.700843] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2270.514395] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2276.037136] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2282.977798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.982333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2282.991958] Lustre: Skipped 1 previous similar message [ 2283.004150] Lustre: Skipped 2 previous similar messages [ 2283.083203] Lustre: server umount lustre-MDT0000 complete [ 2286.562121] LustreError: 64792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187174 with bad export cookie 851608184434125965 [ 2286.570471] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.578081] LustreError: 64792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2288.096431] LustreError: 68565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.097090] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.124275] Lustre: Skipped 1 previous similar message [ 2288.130385] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2288.979445] Lustre: server umount lustre-MDT0001 complete [ 2295.213066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2304.589502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.608126] LustreError: 63721:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c733f567 x1870855679314560/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2312.838861] LustreError: 69301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.921620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.974846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2316.545974] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2324.585098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.004377] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2329.728479] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2335.275765] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2337.451980] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2343.145084] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2349.979523] Lustre: server umount lustre-MDT0000 complete [ 2350.563299] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.567131] LustreError: 66645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.572642] Lustre: Skipped 1 previous similar message [ 2350.588581] LustreError: 66645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2353.361608] LustreError: 64793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187241 with bad export cookie 851608184434127344 [ 2353.368209] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2353.373215] LustreError: 64793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2355.863548] Lustre: server umount lustre-MDT0001 complete [ 2362.882564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2367.985618] Lustre: DEBUG MARKER: files compared the same [ 2370.639729] Lustre: server umount lustre-OST0000 complete [ 2390.797825] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2394.266204] Key type lgssc unregistered [ 2394.641234] LNet: 74532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2394.662056] LNetError: 74532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.690627] LNet: Removed LNI 192.168.204.120@tcp [ 2395.719822] Key type .llcrypt unregistered [ 2395.723086] Key type ._llcrypt unregistered [ 2419.219116] Key type ._llcrypt registered [ 2419.222974] Key type .llcrypt registered [ 2419.336875] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2420.255164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2420.272898] alg: No test for adler32 (adler32-zlib) [ 2421.388943] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 2421.619740] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2423.303753] Key type lgssc registered [ 2424.575716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2432.342112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2432.356269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.746506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.511759] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2444.267409] LustreError: 75952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.291878] LustreError: 75952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2447.335620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.601728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2452.035589] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2459.491982] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2465.525720] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2472.633384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2472.916631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.557718] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2480.114277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2487.894803] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2500.575915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2500.582422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.597265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2502.626710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2502.628632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2502.634268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.661369] Lustre: Skipped 1 previous similar message [ 2506.715941] Lustre: server umount lustre-OST0000 complete [ 2512.866849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.869808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2512.894492] Lustre: Skipped 1 previous similar message [ 2516.731638] Lustre: server umount lustre-MDT0000 complete [ 2517.985333] LustreError: 75947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.007297] LustreError: 75947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2521.099652] LustreError: 75932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187408 with bad export cookie 2462276661302681753 [ 2521.110276] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.469909] Lustre: server umount lustre-MDT0001 complete [ 2532.999390] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2535.835994] Key type lgssc unregistered [ 2536.107699] LNet: 79498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.116528] LNetError: 79498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2536.150548] LNet: Removed LNI 192.168.204.120@tcp [ 2536.968067] Key type .llcrypt unregistered [ 2536.971547] Key type ._llcrypt unregistered [ 2562.138737] Key type ._llcrypt registered [ 2562.143887] Key type .llcrypt registered [ 2562.306744] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2563.531799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2563.746692] alg: No test for adler32 (adler32-zlib) [ 2564.900804] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 2565.274751] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2567.047276] Key type lgssc registered [ 2568.046494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2576.305903] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2576.334926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2592.031385] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2597.151433] LustreError: 80889:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 2623.840409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.049511] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2638.479349] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2645.886350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.039650] Lustre: *** cfs_fail_loc=706, val=0*** [ 2646.446688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.570546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2650.679605] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2651.619097] LustreError: 80915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.641313] LustreError: 80915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2656.747963] LustreError: 80914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.797439] LustreError: 80915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.821653] LustreError: 80915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2659.178810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.501260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2664.365563] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2671.946961] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2677.273969] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2680.298919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2680.301224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.305167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2680.330621] Lustre: Skipped 1 previous similar message [ 2685.411847] LustreError: 80903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.470521] Lustre: server umount lustre-OST0000 complete [ 2690.016645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2690.031613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.045588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.052246] Lustre: Skipped 1 previous similar message [ 2695.650339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.659522] Lustre: Skipped 2 previous similar messages [ 2700.780019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2700.795102] Lustre: Skipped 1 previous similar message [ 2704.351171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2704.558752] Lustre: server umount lustre-MDT0000 complete [ 2705.889308] LustreError: 81974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.900735] LustreError: 81974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2708.859687] LustreError: 80897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187596 with bad export cookie 14979455039257021030 [ 2708.871455] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.880755] LustreError: 80897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2709.180520] Lustre: server umount lustre-MDT0001 complete [ 2720.303571] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2722.884719] Key type lgssc unregistered [ 2723.131735] LNet: 84504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2723.138722] LNetError: 84504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2723.155688] LNet: Removed LNI 192.168.204.120@tcp [ 2724.028835] Key type .llcrypt unregistered [ 2724.034379] Key type ._llcrypt unregistered [ 2750.906683] Key type ._llcrypt registered [ 2750.909861] Key type .llcrypt registered [ 2751.025433] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2752.191162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2752.230775] alg: No test for adler32 (adler32-zlib) [ 2753.324492] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 2753.586455] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2755.295179] Key type lgssc registered [ 2756.427784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2765.079806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2765.099659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.910564] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2779.507209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.100444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.129281] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2785.256048] LustreError: 86611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2785.281803] LustreError: 86611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2789.348763] LustreError: 86615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.377615] LustreError: 86615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2794.468756] LustreError: 86614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2794.484943] LustreError: 86614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2794.814597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.243653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.613670] LustreError: 86615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.641343] LustreError: 86615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2802.444355] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2804.714531] LustreError: 86611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.747166] LustreError: 86611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2811.823553] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.840049] Lustre: MGS: Client f545d0ec-2ec7-4483-8e8a-655e1127d625 (at 0@lo) reconnecting [ 2811.870885] LustreError: 86625:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2812.139565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.216403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2817.455841] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2832.868042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.890799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2832.891420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2832.907992] Lustre: Skipped 1 previous similar message [ 2836.812689] Lustre: server umount lustre-OST0000 complete [ 2843.107152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2843.116823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.131550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.153641] Lustre: Skipped 2 previous similar messages [ 2843.184293] Lustre: Skipped 1 previous similar message [ 2848.224737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.234184] Lustre: Skipped 1 previous similar message [ 2852.322282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.327512] Lustre: Skipped 1 previous similar message [ 2855.391137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2855.531970] Lustre: server umount lustre-MDT0000 complete [ 2857.445645] LustreError: 88156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.469381] LustreError: 88156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2860.911032] Lustre: server umount lustre-MDT0001 complete [ 2865.656687] Lustre: server umount MGS complete [ 2878.528945] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:42:43 (1784187763) [ 2912.316783] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 2921.520283] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2958.965930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.159914] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2975.592788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.848307] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2975.899751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2976.050943] Lustre: lustre-MDT0001: new disk, initializing [ 2976.193528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2976.243607] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2976.251850] Lustre: 93781:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2976.263863] LustreError: 93781:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2981.408656] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2991.657794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2991.865456] Lustre: lustre-OST0000: new disk, initializing [ 2991.869405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2991.875528] Lustre: 94547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2991.932314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.807777] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2998.826286] Lustre: 94900:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000032e2b5a9 x1870856339891840/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784187902 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3007.625139] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3014.712509] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.726951] Lustre: MGS: Client 6020de5b-6fe3-4c2a-b19c-3a65d18f4bf5 (at 0@lo) reconnecting [ 3014.795338] Lustre: 93051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3014.811869] LustreError: 93051:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 3014.817390] LustreError: 93051:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3014.839615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3014.895294] Lustre: lustre-MDT0000: new disk, initializing [ 3015.018714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.041440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3015.068237] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3016.096232] Lustre: 95616:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000fd0b56ad x1870856339898624/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1784187919 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3019.564143] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3021.169236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3021.192109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3021.324057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3045.870922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.872154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3045.885890] Lustre: Skipped 1 previous similar message [ 3045.900913] Lustre: Skipped 1 previous similar message [ 3050.567392] Lustre: server umount lustre-OST0000 complete [ 3056.096921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3056.098590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.104754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3056.114018] Lustre: Skipped 1 previous similar message [ 3056.125853] Lustre: Skipped 1 previous similar message [ 3060.802618] Lustre: server umount lustre-MDT0000 complete [ 3061.221380] LustreError: 93757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3061.235590] LustreError: 93757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3071.009887] Lustre: server umount lustre-MDT0001 complete [ 3075.130464] Lustre: server umount MGS complete [ 3087.326181] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:46:12 (1784187972) [ 3106.087863] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3109.084407] Key type lgssc unregistered [ 3109.437588] LNet: 98116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3109.448268] LNetError: 98116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3109.462296] LNet: Removed LNI 192.168.204.120@tcp [ 3110.136394] Key type .llcrypt unregistered [ 3110.139192] Key type ._llcrypt unregistered [ 3128.760382] Key type ._llcrypt registered [ 3128.763173] Key type .llcrypt registered [ 3128.898654] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3130.411731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3130.527631] alg: No test for adler32 (adler32-zlib) [ 3131.660291] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 3131.975319] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3133.735509] Key type lgssc registered [ 3135.735768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3145.480668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3145.501264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.944732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.542750] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3156.458245] LustreError: 99338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.477817] LustreError: 99338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3159.429317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.823781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3164.187351] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3172.118195] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3178.279324] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3185.150909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.486927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.679103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3192.291128] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3200.740500] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3215.332829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.333763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.335770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.347280] Lustre: Skipped 1 previous similar message [ 3218.954897] Lustre: server umount lustre-OST0000 complete [ 3225.570055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.585456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.590397] Lustre: Skipped 2 previous similar messages [ 3230.690766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.696060] Lustre: Skipped 1 previous similar message [ 3235.810658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.815525] Lustre: Skipped 1 previous similar message [ 3237.344211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3237.537754] Lustre: server umount lustre-MDT0000 complete [ 3240.929151] LustreError: 99339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.952603] LustreError: 99339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3241.352136] LustreError: 99320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188129 with bad export cookie 16745115144404521277 [ 3241.365476] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.369607] LustreError: 99320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3241.678344] Lustre: server umount lustre-MDT0001 complete [ 3254.046723] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3256.460860] Key type lgssc unregistered [ 3256.825782] LNet: 102887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3256.841928] LNetError: 102887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3256.868503] LNet: Removed LNI 192.168.204.120@tcp [ 3257.789772] Key type .llcrypt unregistered [ 3257.791826] Key type ._llcrypt unregistered [ 3275.860259] Key type ._llcrypt registered [ 3275.864373] Key type .llcrypt registered [ 3276.000527] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3277.720950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3277.781335] alg: No test for adler32 (adler32-zlib) [ 3278.861536] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 3279.164090] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3280.903996] Key type lgssc registered [ 3282.491356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3289.892073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3289.899510] LustreError: 103968:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3289.913774] LustreError: 103971:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3289.924150] LustreError: 103968:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3289.935698] LustreError: 103968:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3289.943048] LustreError: 103968:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3289.947123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.424419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.223482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.527494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3305.077514] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3311.328302] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lsmod [ 3314.403915] LustreError: 105175:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3314.409048] LustreError: 105176:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3314.415416] LustreError: 105175:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3314.425143] LustreError: 105175:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3314.434846] LustreError: 105175:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3314.441352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.026078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.056513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3321.312640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3321.322265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.353138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3322.352655] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.358401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3322.411169] Lustre: Skipped 2 previous similar messages [ 3324.205812] Lustre: server umount lustre-MDT0000 complete [ 3325.479356] LustreError: 104009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.509180] LustreError: 104009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3327.850877] LustreError: 103984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188215 with bad export cookie 7906195306257270630 [ 3327.867276] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.867595] LustreError: 103984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3329.695819] LustreError: 104439:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f7812495 x1870856890373760/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3329.936650] Lustre: server umount lustre-MDT0001 complete [ 3344.271178] Lustre: server umount lustre-OST0000 complete [ 3353.387324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.992262] LustreError: 106242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.102584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.171435] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3359.204744] LustreError: 106243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.220342] LustreError: 106243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3363.297295] LustreError: 106242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.109111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.497884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3373.501794] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3381.616472] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3387.509694] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3394.161298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3394.519035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.662763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3401.284832] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3409.730273] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3422.184322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3422.190284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.211103] Lustre: Skipped 1 previous similar message [ 3422.222143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.233800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.248795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.263852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.831688] Lustre: server umount lustre-OST0000 complete [ 3434.464256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.480542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3434.488045] Lustre: Skipped 2 previous similar messages [ 3444.705364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.716519] Lustre: Skipped 3 previous similar messages [ 3447.263224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3447.555485] Lustre: server umount lustre-MDT0000 complete [ 3449.833474] LustreError: 108284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.858610] LustreError: 108284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3451.920829] LustreError: 106224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188339 with bad export cookie 7906195306257271575 [ 3451.933045] LustreError: 106224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3451.940809] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.455119] Lustre: server umount lustre-MDT0001 complete [ 3465.223446] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3468.501782] Key type lgssc unregistered [ 3468.839342] LNet: 109781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3468.858138] LNetError: 109781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3468.882797] LNet: Removed LNI 192.168.204.120@tcp [ 3469.948594] Key type .llcrypt unregistered [ 3469.950935] Key type ._llcrypt unregistered [ 3497.051920] Key type ._llcrypt registered [ 3497.060391] Key type .llcrypt registered [ 3497.196478] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3498.936189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.029078] alg: No test for adler32 (adler32-zlib) [ 3500.207436] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 3500.483675] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3502.215393] Key type lgssc registered [ 3503.730493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3513.179091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3513.226994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.862379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.288845] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3525.104759] LustreError: 111200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.120921] LustreError: 111200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3530.213600] LustreError: 111201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3530.513358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.011677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3536.650151] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3546.475937] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3553.655762] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3561.481700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3562.127374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.352509] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3578.489270] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3584.631368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3589.752342] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3594.003283] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3600.356134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.357819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3600.373643] Lustre: Skipped 1 previous similar message [ 3600.383054] Lustre: Skipped 1 previous similar message [ 3604.937882] Lustre: server umount lustre-OST0000 complete [ 3610.593288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.604899] Lustre: Skipped 1 previous similar message [ 3610.606742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.627637] Lustre: Skipped 1 previous similar message [ 3615.712426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3615.730159] Lustre: Skipped 1 previous similar message [ 3620.837409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.844479] Lustre: Skipped 1 previous similar message [ 3623.393830] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3623.692709] Lustre: server umount lustre-MDT0000 complete [ 3625.952909] LustreError: 111197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.972872] LustreError: 111197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3628.094212] LustreError: 111181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188515 with bad export cookie 3012976510439148952 [ 3628.102117] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3628.113083] LustreError: 111181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3628.507137] Lustre: server umount lustre-MDT0001 complete [ 3641.826326] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3646.113165] Key type lgssc unregistered [ 3646.554519] LNet: 114858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3646.569737] LNetError: 114858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3647.662760] LNet: Removed LNI 192.168.204.120@tcp [ 3648.825216] Key type .llcrypt unregistered [ 3648.829222] Key type ._llcrypt unregistered [ 3670.885575] Key type ._llcrypt registered [ 3670.898968] Key type .llcrypt registered [ 3671.049443] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3672.179361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3672.203929] alg: No test for adler32 (adler32-zlib) [ 3673.369926] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 3673.655884] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3675.351326] Key type lgssc registered [ 3676.776743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3687.104261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3687.121176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.605619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.065379] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3699.176345] LustreError: 116097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.205192] LustreError: 116097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3704.306349] LustreError: 116098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.437886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.979570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3710.552946] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3719.431708] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3727.815283] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3735.569588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3736.109575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.357799] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3752.170963] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3758.718386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3774.432320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3774.442666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.459472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3774.476225] Lustre: Skipped 1 previous similar message [ 3774.490621] Lustre: Skipped 1 previous similar message [ 3777.749706] Lustre: server umount lustre-OST0000 complete [ 3784.672497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.709784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.727590] Lustre: Skipped 1 previous similar message [ 3788.854241] Lustre: server umount lustre-MDT0000 complete [ 3789.795503] LustreError: 116097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3789.819021] LustreError: 116097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3794.386061] LustreError: 116077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188682 with bad export cookie 871408642980660880 [ 3794.399923] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.404210] LustreError: 116077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3795.096194] Lustre: server umount lustre-MDT0001 complete [ 3808.642199] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3812.338589] Key type lgssc unregistered [ 3812.720106] LNet: 119656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3812.740287] LNetError: 119656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3812.776077] LNet: Removed LNI 192.168.204.120@tcp [ 3813.692521] Key type .llcrypt unregistered [ 3813.695429] Key type ._llcrypt unregistered [ 3841.194774] Key type ._llcrypt registered [ 3841.197674] Key type .llcrypt registered [ 3841.322777] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3842.833379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3842.873089] alg: No test for adler32 (adler32-zlib) [ 3843.892944] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 3844.147869] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3845.807241] Key type lgssc registered [ 3847.094069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3855.964734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3855.995714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.662772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.181355] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3868.134336] LustreError: 121074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.179642] LustreError: 121074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3873.254098] LustreError: 121073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.397931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.950868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3878.694939] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3886.932495] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3893.340557] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3900.613190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3900.989555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.053465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3908.355549] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3916.747947] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3924.283186] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3928.292241] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3928.296796] Lustre: Skipped 2 previous similar messages [ 3933.103791] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3933.120262] Lustre: Skipped 2 previous similar messages [ 3934.277297] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3934.313253] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3939.502556] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.20@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3939.510805] Lustre: Skipped 2 previous similar messages [ 3940.597641] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.20@tcp 192.168.0.[2,10]@tcp [ 3954.878231] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.20/24@tcp in log lustre-MDT0000 [ 3954.891393] Lustre: Skipped 2 previous similar messages [ 3960.898796] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.20/24@tcp [ 3960.905990] Lustre: Skipped 1 previous similar message [ 3969.507130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3969.507665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.509534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3969.571030] Lustre: Skipped 1 previous similar message [ 3974.624522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3974.631583] Lustre: Skipped 2 previous similar messages [ 3975.437598] Lustre: server umount lustre-OST0000 complete [ 3981.283338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3981.293965] LustreError: Skipped 1 previous similar message [ 3981.301208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.317078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3984.865294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3984.868367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.839832] Lustre: server umount lustre-MDT0000 complete [ 3989.984637] LustreError: 121074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3989.996510] LustreError: 121074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3992.135420] LustreError: 121054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188879 with bad export cookie 2199002777595451906 [ 3992.142643] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3992.159621] LustreError: 121054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3992.958150] Lustre: server umount lustre-MDT0001 complete [ 4005.784602] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4008.601299] Key type lgssc unregistered [ 4008.989792] LNet: 125793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4008.998049] LNetError: 125793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4009.022672] LNet: Removed LNI 192.168.204.120@tcp [ 4010.050571] Key type .llcrypt unregistered [ 4010.054123] Key type ._llcrypt unregistered [ 4042.187822] Key type ._llcrypt registered [ 4042.191263] Key type .llcrypt registered [ 4042.295491] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4043.220927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4043.239238] alg: No test for adler32 (adler32-zlib) [ 4044.299274] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 4044.488845] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4046.153702] Key type lgssc registered [ 4047.677490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4055.331048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4055.361551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.826795] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4056.849815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4056.860734] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.20/24@tcp [ 4061.510308] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4066.279503] LustreError: 127355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.308144] LustreError: 127355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4070.347028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.810431] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4070.844503] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.20/24@tcp [ 4070.895307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4076.369295] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4085.704792] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4093.232275] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4100.779872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4101.044550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.088454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4108.385671] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4116.432385] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4127.713806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4127.722827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4127.742121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4130.787074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.791601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4130.825794] Lustre: Skipped 1 previous similar message [ 4133.295110] Lustre: server umount lustre-OST0000 complete [ 4137.441105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4137.449842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.470061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.024898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.025703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.707135] Lustre: server umount lustre-MDT0000 complete [ 4146.149558] LustreError: 128799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.172816] LustreError: 128799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4148.055151] LustreError: 127336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189035 with bad export cookie 9893339116277904931 [ 4148.058926] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4148.074786] LustreError: 127336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4148.538839] Lustre: server umount lustre-MDT0001 complete [ 4161.148638] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4163.823738] Key type lgssc unregistered [ 4164.144895] LNet: 130944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4164.159115] LNetError: 130944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4164.186125] LNet: Removed LNI 192.168.204.120@tcp [ 4164.931664] Key type .llcrypt unregistered [ 4164.934343] Key type ._llcrypt unregistered [ 4192.241408] Key type ._llcrypt registered [ 4192.247596] Key type .llcrypt registered [ 4192.384917] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4208.367879] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4209.534474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4209.675084] alg: No test for adler32 (adler32-zlib) [ 4210.744977] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 4210.994429] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4212.655240] Key type lgssc registered [ 4213.902776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4253.824246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4253.859168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.136362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4255.185690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4255.260322] Lustre: lustre-MDT0000: new disk, initializing [ 4255.325258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.346708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4260.652981] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4271.744109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.857221] Lustre: 134347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4271.915576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4271.919373] Lustre: Skipped 1 previous similar message [ 4271.987898] Lustre: lustre-MDT0001: new disk, initializing [ 4272.105084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4272.151819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4272.161458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4277.496786] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4288.243760] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4294.914491] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4303.872501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4304.295625] Lustre: lustre-OST0000: new disk, initializing [ 4304.298195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4304.315779] Lustre: 136316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4304.501813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.618272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4309.633203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4309.712980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4311.196820] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4322.351739] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4327.880491] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4328.098653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4332.625952] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4332.829602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4335.586899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.587649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4335.592942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4335.608825] Lustre: Skipped 1 previous similar message [ 4340.710244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.715572] Lustre: Skipped 2 previous similar messages [ 4341.207939] Lustre: server umount lustre-OST0000 complete [ 4345.827850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.840754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4345.846866] Lustre: Skipped 1 previous similar message [ 4350.984969] Lustre: server umount lustre-MDT0000 complete [ 4355.380870] LustreError: 135140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189243 with bad export cookie 4848651325568183278 [ 4355.449740] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4355.923843] Lustre: server umount lustre-MDT0001 complete [ 4366.407576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.790907] LustreError: 138531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.804950] LustreError: 138531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4366.839179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.608260] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4371.944357] LustreError: 138532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.064209] LustreError: 138532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4381.629628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4381.927429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4386.942365] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4395.332564] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4401.956489] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4409.052900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4409.429668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.439986] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4425.024426] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4430.123985] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4430.346246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4435.328739] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4435.649355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4444.692564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4444.891449] Lustre: lustre-OST0001: new disk, initializing [ 4444.897407] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4444.904557] Lustre: 141904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4445.012500] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4452.560777] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4452.964122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4452.978481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4453.049688] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4461.513514] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4461.744258] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4467.101646] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4467.360885] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4474.533761] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f7498219800.ost_server_uuid 50 [ 4476.568900] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f7498219800.ost_server_uuid in IDLE state after 0 sec [ 4489.184791] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4489.198847] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.212949] LustreError: Skipped 1 previous similar message [ 4489.213967] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4489.224379] Lustre: Skipped 2 previous similar messages [ 4489.238323] Lustre: Skipped 1 previous similar message [ 4489.450432] Lustre: server umount lustre-OST0001 complete [ 4494.310363] LustreError: 142288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.317647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.336179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4494.336562] LustreError: 142288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4494.363500] Lustre: Skipped 1 previous similar message [ 4499.439790] LustreError: 143498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.455110] LustreError: 143498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4500.353090] Lustre: server umount lustre-OST0000 complete [ 4505.057381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4505.082135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.102331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4505.129952] Lustre: Skipped 3 previous similar messages [ 4511.284884] Lustre: server umount lustre-MDT0000 complete [ 4514.787024] LustreError: 138526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.796467] LustreError: 138526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4516.241372] LustreError: 138512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189404 with bad export cookie 4848651325568184160 [ 4516.244534] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4516.278820] LustreError: 138512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4517.192593] Lustre: server umount lustre-MDT0001 complete [ 4540.130882] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4542.966721] Key type lgssc unregistered [ 4543.350955] LNet: 144910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4543.357450] LNetError: 144910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4543.365672] LNet: Removed LNI 192.168.204.120@tcp [ 4544.395075] Key type .llcrypt unregistered [ 4544.401481] Key type ._llcrypt unregistered [ 4589.901535] Key type ._llcrypt registered [ 4589.912083] Key type .llcrypt registered [ 4590.087212] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4606.789788] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4608.421165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4608.492104] alg: No test for adler32 (adler32-zlib) [ 4609.652772] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 4609.913927] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4611.647177] Key type lgssc registered [ 4613.748606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4658.373043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4658.407626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.699951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4659.730519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4659.801116] Lustre: lustre-MDT0000: new disk, initializing [ 4659.899424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.925559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4665.448754] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4677.659623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.794433] Lustre: 149069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4677.856072] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4677.862553] Lustre: Skipped 1 previous similar message [ 4677.994617] Lustre: lustre-MDT0001: new disk, initializing [ 4678.093290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4678.140565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4678.165470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4683.619863] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4694.867682] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4702.344651] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4712.407909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4712.787522] Lustre: lustre-OST0000: new disk, initializing [ 4712.792495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4712.805336] Lustre: 151039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4712.914573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.249966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4721.269698] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4721.503325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4721.532512] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4733.572137] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4737.965325] Lustre: Failing over lustre-OST0000 [ 4738.169476] Lustre: server umount lustre-OST0000 complete [ 4755.343109] Lustre: 147290:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784189627/real 1784189627] req@000000007b3d2f55 x1870858286042496/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1784189643 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4755.396118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.471315] Lustre: 147291:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784189629/real 1784189629] req@0000000042e5e3a2 x1870858286043648/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784189645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4757.532859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4758.624101] Lustre: 147290:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784189629/real 1784189629] req@00000000ab38abf4 x1870858286043392/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784189645 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4758.671624] Lustre: 147290:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4759.848200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4760.555705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4760.579447] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4763.564176] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784189635/real 1784189635] req@00000000553138c7 x1870858286044672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784189651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4765.676809] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4767.857562] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4767.858342] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4768.532886] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4768.864229] Lustre: 147289:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784189640/real 1784189640] req@00000000f84e54a9 x1870858286045952/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784189656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4768.884796] Lustre: 147289:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4772.874150] Lustre: Failing over lustre-MDT0000 [ 4773.122451] Lustre: server umount lustre-MDT0000 complete [ 4775.392099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4775.404101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.029547] LustreError: 150091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4781.060249] LustreError: 150091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4782.496689] LustreError: 149076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.145934] LustreError: 149079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.163145] LustreError: 149079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4791.263164] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784189663/real 1784189663] req@000000000a80ef63 x1870858286058368/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784189679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4791.266928] LustreError: 149080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.300633] Lustre: 147292:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4791.309611] LustreError: 149080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4791.335936] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4793.568371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.387243] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4796.422211] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4801.505027] LustreError: 147288:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000240be1dc x1870858286063616/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4801.880426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.941898] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4804.005600] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4807.165089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4807.172372] Lustre: Skipped 1 previous similar message [ 4807.196371] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4807.261832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4808.066531] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4817.889524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4817.912933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.938967] Lustre: Skipped 2 previous similar messages [ 4817.943715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4819.429894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4819.447354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4819.771215] Lustre: server umount lustre-OST0000 complete [ 4827.625804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4827.627091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4827.631254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4827.650887] Lustre: Skipped 2 previous similar messages [ 4830.968228] Lustre: server umount lustre-MDT0000 complete [ 4832.742447] LustreError: 149074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4832.791795] LustreError: 149074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4835.206755] LustreError: 149061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189722 with bad export cookie 12842555007615523816 [ 4835.208293] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.215897] LustreError: 149061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4835.819196] Lustre: server umount lustre-MDT0001 complete [ 4849.929779] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4854.932675] Key type lgssc unregistered [ 4855.414351] LNet: 154698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4855.427848] LNetError: 154698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4855.455844] LNet: Removed LNI 192.168.204.120@tcp [ 4856.886224] Key type .llcrypt unregistered [ 4856.888589] Key type ._llcrypt unregistered [ 4898.395632] Key type ._llcrypt registered [ 4898.397723] Key type .llcrypt registered [ 4898.522588] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4899.734995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4899.849519] alg: No test for adler32 (adler32-zlib) [ 4901.142701] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 4901.573548] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4903.312229] Key type lgssc registered [ 4904.733322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4914.740434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4914.775578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4916.344565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4922.357464] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4926.949859] LustreError: 156219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.981955] LustreError: 156219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4932.095596] LustreError: 156220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4933.759170] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4937.189859] LustreError: 156219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4937.935292] LustreError: 156248:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4937.944213] LustreError: 156248:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4937.951214] LustreError: 156248:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4938.193801] Lustre: server umount lustre-MDT0000 complete [ 4960.114746] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4973.910083] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5029.064190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5029.572384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5029.640038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5029.815369] Lustre: lustre-MDT0000: new disk, initializing [ 5030.003383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.059847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5037.121430] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5049.697902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5049.838754] Lustre: 159559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5049.900244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5049.904729] Lustre: Skipped 1 previous similar message [ 5049.970387] Lustre: lustre-MDT0001: new disk, initializing [ 5050.075879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5050.114647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5050.125326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5055.262666] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5066.107860] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5073.145587] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5081.109276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5081.491696] Lustre: lustre-OST0000: new disk, initializing [ 5081.498313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5081.504844] Lustre: 161527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5081.617279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.118669] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5091.447109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5091.459119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5091.594804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5100.833034] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5106.001436] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5108.342473] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:19:53 (1784189993) [ 5413.856047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5413.868645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.905395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5414.376148] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.387740] Lustre: Skipped 1 previous similar message [ 5419.397427] Lustre: server umount lustre-MDT0000 complete [ 5419.488796] LustreError: 159570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.424394] LustreError: 159550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190311 with bad export cookie 10953018437443226322 [ 5423.428176] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5423.431327] LustreError: 159550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5424.608364] LustreError: 159564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5424.609122] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.639778] LustreError: 159564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5424.655365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5424.671540] Lustre: Skipped 3 previous similar messages [ 5428.640224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5429.855667] Lustre: server umount lustre-MDT0001 complete [ 5440.462819] Lustre: server umount lustre-OST0000 complete [ 5451.326507] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5462.224936] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5503.239897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.714958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5503.794707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5503.946326] Lustre: lustre-MDT0000: new disk, initializing [ 5504.083651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5504.105857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5509.658315] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5521.399366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.611792] Lustre: 166200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5521.674752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5521.682878] Lustre: Skipped 1 previous similar message [ 5521.820684] Lustre: lustre-MDT0001: new disk, initializing [ 5521.994496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5522.061566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5522.078135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5526.478399] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5536.704877] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5542.588042] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5550.835232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5551.287082] Lustre: lustre-OST0000: new disk, initializing [ 5551.296466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5551.309194] Lustre: 168171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5551.476610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.647581] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5561.414130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5561.438767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5561.586317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5570.636249] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5577.083501] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5577.455475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5584.127791] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5584.383836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5587.432743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.437755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.449244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.473285] Lustre: Skipped 1 previous similar message [ 5592.545425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5592.567199] Lustre: Skipped 2 previous similar messages [ 5601.247226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5601.549293] Lustre: server umount lustre-OST0000 complete [ 5607.905177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.926895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5607.938746] Lustre: Skipped 3 previous similar messages [ 5612.623680] Lustre: server umount lustre-MDT0000 complete [ 5613.026621] LustreError: 166209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5613.058043] LustreError: 166209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5617.265974] LustreError: 166193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190505 with bad export cookie 10953018437443385152 [ 5617.283852] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5617.292035] LustreError: 166193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5617.689723] Lustre: server umount lustre-MDT0001 complete [ 5627.580801] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:28:32 (1784190512) [ 5645.659461] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5656.318870] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5698.781762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.077796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5699.197393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5699.341480] Lustre: lustre-MDT0000: new disk, initializing [ 5699.515430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.568291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5704.075456] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5715.011419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5715.232590] Lustre: 172931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5715.659823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5719.877897] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5729.272561] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5735.235330] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5742.516607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5742.833540] Lustre: lustre-OST0000: new disk, initializing [ 5742.836484] Lustre: Skipped 1 previous similar message [ 5742.839628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5742.843347] Lustre: Skipped 2 previous similar messages [ 5742.848803] Lustre: 174907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5749.816372] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5751.351865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5751.370757] Lustre: Skipped 1 previous similar message [ 5751.381710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5751.504134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5760.603906] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5772.260783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.261530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5772.277356] Lustre: Skipped 1 previous similar message [ 5772.282100] LustreError: Skipped 1 previous similar message [ 5772.302943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5772.316524] Lustre: Skipped 1 previous similar message [ 5774.375799] Lustre: server umount lustre-OST0000 complete [ 5781.992572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5782.001589] LustreError: Skipped 1 previous similar message [ 5784.701241] Lustre: server umount lustre-MDT0000 complete [ 5787.630467] LustreError: 172943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.648603] LustreError: 172943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5788.445283] LustreError: 172923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190676 with bad export cookie 10953018437443386034 [ 5788.454931] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.456876] LustreError: 172923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5788.920699] Lustre: server umount lustre-MDT0001 complete [ 5798.812968] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:31:23 (1784190683) [ 5818.229750] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5829.598607] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5868.255639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.650811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5868.697608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5868.764651] Lustre: lustre-MDT0000: new disk, initializing [ 5868.855495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.861576] Lustre: Skipped 2 previous similar messages [ 5868.876627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5874.022427] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5885.957817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.090853] Lustre: 179236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5886.316119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5891.595160] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5902.467228] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5909.704900] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5917.257467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.599966] Lustre: 181211:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5923.902616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5924.055891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5925.055282] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5935.427262] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5944.805902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.809087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5944.820655] Lustre: Skipped 3 previous similar messages [ 5944.849792] Lustre: Skipped 4 previous similar messages [ 5950.513916] Lustre: server umount lustre-OST0000 complete [ 5958.112968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5961.574505] Lustre: server umount lustre-MDT0000 complete [ 5965.281389] LustreError: 179243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.304216] LustreError: 179243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5965.761581] LustreError: 179227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190853 with bad export cookie 10953018437443387119 [ 5965.764143] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5965.778604] LustreError: 179227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5966.438855] Lustre: server umount lustre-MDT0001 complete [ 5978.436591] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5981.731016] Key type lgssc unregistered [ 5982.160064] LNet: 183067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5982.178863] LNetError: 183067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5982.197941] LNet: Removed LNI 192.168.204.120@tcp [ 5983.241111] Key type .llcrypt unregistered [ 5983.243799] Key type ._llcrypt unregistered [ 6011.926763] Key type ._llcrypt registered [ 6011.930729] Key type .llcrypt registered [ 6012.103971] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6013.577687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6013.649525] alg: No test for adler32 (adler32-zlib) [ 6014.774746] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 6015.045595] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6016.759278] Key type lgssc registered [ 6018.033390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6026.262208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6026.296139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.702114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6032.502808] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6037.991287] LustreError: 184487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.013216] LustreError: 184487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6040.414753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.788423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6045.737023] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6053.282351] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6060.068876] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6067.528136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6067.989931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.151242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6075.362052] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6084.737234] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6104.034521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6104.048856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.064790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6106.086502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6106.096609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6106.122190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.528357] Lustre: server umount lustre-OST0000 complete [ 6116.320778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.354974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6116.373758] Lustre: Skipped 2 previous similar messages [ 6120.780297] Lustre: server umount lustre-MDT0000 complete [ 6121.442656] LustreError: 184488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6121.465783] LustreError: 184488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6125.132083] LustreError: 184468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191012 with bad export cookie 5321773221749878090 [ 6125.141071] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6125.489962] Lustre: server umount lustre-MDT0001 complete [ 6140.108076] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6143.193492] Key type lgssc unregistered [ 6143.507718] LNet: 188039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6143.515193] LNetError: 188039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6143.530641] LNet: Removed LNI 192.168.204.120@tcp [ 6144.423580] Key type .llcrypt unregistered [ 6144.431551] Key type ._llcrypt unregistered [ 6172.860416] Key type ._llcrypt registered [ 6172.863451] Key type .llcrypt registered [ 6172.962556] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6174.316712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6174.342959] alg: No test for adler32 (adler32-zlib) [ 6175.460883] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 6175.708355] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6177.383229] Key type lgssc registered [ 6178.626844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6187.109142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6187.127272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6188.596792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6193.908825] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6198.769835] LustreError: 189465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.805473] LustreError: 189465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6202.806279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6203.325340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6208.296986] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6217.645161] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6224.519045] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6231.966051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6232.501208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6233.541639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6239.973291] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6249.676662] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6257.611361] Lustre: lustre-OST0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 6258.153934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6258.161915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6258.165942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6258.197032] Lustre: Skipped 1 previous similar message [ 6262.703538] Lustre: lustre-OST0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 6262.715572] Lustre: Skipped 1 previous similar message [ 6263.611352] Lustre: server umount lustre-OST0000 complete [ 6272.818186] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f7498d31800.ost_server_uuid 50 [ 6274.572537] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6288.864334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.869679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6288.874450] Lustre: Skipped 1 previous similar message [ 6288.888766] Lustre: Skipped 3 previous similar messages [ 6294.944620] Lustre: server umount lustre-MDT0000 complete [ 6295.010669] LustreError: 189465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6295.023910] LustreError: 189465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6299.248209] LustreError: 189446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191187 with bad export cookie 913037941444443184 [ 6299.263526] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.582361] Lustre: server umount lustre-MDT0001 complete [ 6308.918168] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:39:54 (1784191194) [ 6316.546603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.956680] LustreError: 193384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.001867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.150587] LustreError: 193385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.982837] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6326.242660] LustreError: 193384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.169709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.629220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6338.108180] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6345.683067] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6351.381505] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6358.097515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6358.450176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6361.540815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6365.527212] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6373.412737] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6380.585593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6380.879562] Lustre: lustre-OST0001: new disk, initializing [ 6380.886769] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6380.895041] Lustre: 196227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6380.984731] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6389.261608] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6390.920404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6390.928780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6391.140363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6401.841077] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6405.602112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6405.618061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6405.641312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6405.644543] Lustre: Skipped 2 previous similar messages [ 6411.002157] Lustre: server umount lustre-OST0000 complete [ 6411.760633] LustreError: 195205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6411.778030] LustreError: 195205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6416.817947] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6416.992850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6420.893602] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6421.148893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6428.593757] LustreError: 195207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6428.617496] LustreError: 195207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6437.351066] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6437.373592] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.384540] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6437.411772] Lustre: Skipped 2 previous similar messages [ 6437.425905] Lustre: Skipped 4 previous similar messages [ 6442.030654] Lustre: server umount lustre-OST0001 complete [ 6446.564231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6446.588654] LustreError: Skipped 1 previous similar message [ 6446.602158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.108386] Lustre: server umount lustre-MDT0000 complete [ 6457.327196] LustreError: 196226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191345 with bad export cookie 913037941444444255 [ 6457.330777] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6457.340800] LustreError: 196226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6457.729180] Lustre: server umount lustre-MDT0001 complete [ 6484.639846] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:42:49 (1784191369) [ 6492.348143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6492.737513] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6492.781966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6493.029304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6498.325224] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6508.185483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.384712] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6508.447226] Lustre: 199666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6514.029834] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6522.203103] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6529.123983] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6535.862756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6536.065990] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6544.008252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6544.272456] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6552.768109] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6559.971844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6560.064395] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6560.376437] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6560.389500] Lustre: Skipped 2 previous similar messages [ 6567.287949] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6569.617920] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6576.751148] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6579.682397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.693109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.712804] Lustre: Skipped 1 previous similar message [ 6579.718250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6579.726079] Lustre: Skipped 5 previous similar messages [ 6585.864398] Lustre: server umount lustre-OST0000 complete [ 6587.810126] LustreError: 202563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.853360] LustreError: 202563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6605.284534] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6605.293409] LustreError: Skipped 1 previous similar message [ 6605.300068] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.314502] Lustre: Skipped 1 previous similar message [ 6608.000833] Lustre: server umount lustre-OST0001 complete [ 6616.039456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.057200] Lustre: Skipped 8 previous similar messages [ 6626.271168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6626.453581] Lustre: server umount lustre-MDT0000 complete [ 6630.540995] LustreError: 199660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191518 with bad export cookie 913037941444445662 [ 6630.552076] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.560976] LustreError: 199660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6630.985332] Lustre: server umount lustre-MDT0001 complete [ 6656.534246] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:45:41 (1784191541) [ 6673.947450] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6684.860469] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6727.049052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.299277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6727.338248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6727.553821] Lustre: lustre-MDT0000: new disk, initializing [ 6727.681908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6727.707813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6732.919502] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6744.093780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.215434] Lustre: 207768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6744.264548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6744.268448] Lustre: Skipped 1 previous similar message [ 6744.329195] Lustre: lustre-MDT0001: new disk, initializing [ 6744.428712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6744.453619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6749.565665] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6760.989115] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6768.143969] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6776.390650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6776.698523] Lustre: lustre-OST0000: new disk, initializing [ 6776.704782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6776.713479] Lustre: 209737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6782.506548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6782.514949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6782.620936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6783.689859] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6795.200727] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6800.549378] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6800.798062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6805.900777] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6806.381852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6813.679351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.691830] Lustre: Skipped 4 previous similar messages [ 6813.696437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6813.702597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6813.705515] LustreError: Skipped 2 previous similar messages [ 6813.729243] Lustre: Skipped 5 previous similar messages [ 6823.391733] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6823.622000] Lustre: server umount lustre-OST0000 complete [ 6834.502480] Lustre: server umount lustre-MDT0000 complete [ 6838.668333] LustreError: 207760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191726 with bad export cookie 913037941444447440 [ 6838.672678] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6838.679818] LustreError: 207760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6839.005698] Lustre: server umount lustre-MDT0001 complete [ 6846.746039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.360399] LustreError: 211957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.401921] LustreError: 211957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6853.780802] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6863.860672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6870.339485] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6879.512326] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6887.181115] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6895.666863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.513504] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6912.406242] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6917.364992] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6918.666452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6923.733596] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6923.937388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6927.845859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6927.852168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.868017] Lustre: Skipped 3 previous similar messages [ 6940.127137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6940.351859] Lustre: server umount lustre-OST0000 complete [ 6947.210457] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6951.313426] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6981.088221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6989.112805] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6992.748258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6992.763862] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6992.767504] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6992.792405] Lustre: Skipped 1 previous similar message [ 6998.618395] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7000.598751] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7008.224540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7008.244216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.254969] Lustre: Skipped 12 previous similar messages [ 7012.798657] Lustre: server umount lustre-OST0000 complete [ 7023.778830] Lustre: server umount lustre-MDT0000 complete [ 7028.255877] LustreError: 211937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191916 with bad export cookie 913037941444448329 [ 7028.288738] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7028.966502] Lustre: server umount lustre-MDT0001 complete [ 7039.098279] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:52:04 (1784191924) [ 7046.893249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7047.322221] LustreError: 217933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7047.342716] LustreError: 217933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7047.399218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.413686] Lustre: Skipped 6 previous similar messages [ 7052.518890] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7061.487552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.564660] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7074.706796] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7080.599378] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7087.754135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7095.300364] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7104.926158] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7110.600708] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7110.769637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7115.839972] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7116.236384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7127.744638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7127.956060] Lustre: lustre-OST0001: new disk, initializing [ 7127.964436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7127.972109] Lustre: 221308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7137.396360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7137.414460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7137.595771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7137.769096] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7148.921341] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7154.194585] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7154.456474] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7159.939311] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7160.281553] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7163.362458] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7163.366378] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7163.368471] LustreError: Skipped 3 previous similar messages [ 7163.403211] Lustre: Skipped 5 previous similar messages [ 7168.997499] Lustre: server umount lustre-OST0001 complete [ 7176.496317] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7176.799989] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7182.076948] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7182.274331] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7211.327318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.621648] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7216.644821] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7216.654381] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7216.689716] Lustre: Skipped 1 previous similar message [ 7219.002948] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7229.109030] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7231.077631] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7239.657225] Lustre: server umount lustre-OST0001 complete [ 7263.481702] Lustre: server umount lustre-OST0000 complete [ 7267.815680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.823316] Lustre: Skipped 15 previous similar messages [ 7282.143349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7282.429798] Lustre: server umount lustre-MDT0000 complete [ 7287.560549] LustreError: 217914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192175 with bad export cookie 913037941444449295 [ 7287.573654] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7287.591192] LustreError: 217914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7294.094787] Lustre: server umount lustre-MDT0001 complete [ 7323.552833] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:56:47 (1784192207) [ 7334.227219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.858695] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7334.944761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7341.077625] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7350.955689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.053445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7351.084345] Lustre: 226557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7357.103101] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7366.212300] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7372.161092] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7379.811598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7379.994234] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7387.339624] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7395.346900] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7406.332342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7406.417316] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7414.689407] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7423.546169] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7425.428822] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7430.936273] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7431.216333] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7437.050789] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7437.346913] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7444.674472] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f7483f79800.ost_server_uuid 50 [ 7447.063542] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f7483f79800.ost_server_uuid in IDLE state after 0 sec [ 7448.102271] Lustre: Permanently deactivating lustre-OST0001 [ 7448.128064] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7453.243201] Lustre: Permanently reactivating lustre-OST0001 [ 7453.276747] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7453.283203] Lustre: Skipped 2 previous similar messages [ 7458.805543] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7458.813030] LustreError: Skipped 3 previous similar messages [ 7458.818915] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7458.834716] Lustre: Skipped 6 previous similar messages [ 7464.625895] Lustre: server umount lustre-OST0001 complete [ 7467.494044] LustreError: 228402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7467.510283] LustreError: 228402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7483.359133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7483.648067] Lustre: server umount lustre-OST0000 complete [ 7502.305675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7502.556780] Lustre: server umount lustre-MDT0000 complete [ 7506.906988] LustreError: 226549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192394 with bad export cookie 913037941444450583 [ 7506.916375] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7506.924702] LustreError: 226549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7507.356570] Lustre: server umount lustre-MDT0001 complete [ 7533.846604] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:00:19 (1784192419) [ 7541.759538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7542.042632] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7542.080604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7542.088559] Lustre: Skipped 2 previous similar messages [ 7546.263595] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7554.696490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.812716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7554.836286] Lustre: 233256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7560.336421] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7570.118207] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7577.520948] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7585.737939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7585.979902] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7586.002383] Lustre: Permanently deactivating lustre-OST0000 [ 7586.016914] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7586.257827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.266322] Lustre: Skipped 10 previous similar messages [ 7592.921316] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7594.581514] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7600.729399] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7609.127192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7609.274354] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7617.842664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7618.517968] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7628.019409] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7631.277317] Lustre: Permanently reactivating lustre-OST0000 [ 7631.290382] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7631.293834] Lustre: Skipped 2 previous similar messages [ 7639.469769] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 00000000576644e3, cur 1784192527 deadline 1784192512 last 1784192482 [ 7640.742774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7658.975152] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7659.241740] Lustre: server umount lustre-OST0001 complete [ 7670.095672] Lustre: server umount lustre-OST0000 complete [ 7689.465715] Lustre: server umount lustre-MDT0000 complete [ 7693.598566] LustreError: 233250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192581 with bad export cookie 913037941444452249 [ 7693.616966] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7693.620407] LustreError: 233250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7694.042280] Lustre: server umount lustre-MDT0001 complete [ 7706.999848] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7710.531649] Key type lgssc unregistered [ 7710.900191] LNet: 238152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7710.906291] LNetError: 238152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7710.919367] LNet: Removed LNI 192.168.204.120@tcp [ 7712.027163] Key type .llcrypt unregistered [ 7712.029860] Key type ._llcrypt unregistered [ 7736.113956] Key type ._llcrypt registered [ 7736.118424] Key type .llcrypt registered [ 7736.262227] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7737.149512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7737.309017] alg: No test for adler32 (adler32-zlib) [ 7738.381400] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 7738.637834] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7740.303183] Key type lgssc registered [ 7741.459859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7752.321751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7752.347860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.865900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7758.631727] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7763.427871] LustreError: 239620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7763.447809] LustreError: 239620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7768.098535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.348758] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7768.404672] Lustre: 239609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7768.422898] Lustre: Permanently deactivating lustre-MDT0001 [ 7768.440828] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7768.545068] LustreError: 239619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.580835] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7768.607302] LustreError: 239650:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7768.625566] LustreError: 239650:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7768.649739] LustreError: 239650:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7768.916340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7774.371069] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7783.451038] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7789.031432] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7796.861049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7797.687937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.698396] LustreError: 241460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7798.710750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7803.897144] LustreError: 241461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.464614] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7809.007109] LustreError: 241460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7817.232274] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7819.263933] LustreError: 241461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7819.288509] LustreError: 241461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7825.776803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7825.971626] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7831.057681] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7831.058271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7833.864801] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7843.269261] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7848.031932] Lustre: Permanently reactivating lustre-MDT0001 [ 7848.090101] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7848.092949] Lustre: Skipped 2 previous similar messages [ 7850.307797] Lustre: lustre-MDT0001: Client 0414ae2a-7a99-4fba-b757-c0f58f9659d0 (at 192.168.204.20@tcp) reconnecting [ 7856.824110] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7858.922662] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7865.262917] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7865.550838] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7871.306463] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7871.559656] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7874.412454] Lustre: Permanently deactivating lustre-MDT0001 [ 7874.446490] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7874.454807] Lustre: Skipped 2 previous similar messages [ 7882.901333] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7897.057335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7897.078231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.119413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7897.584326] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.601400] Lustre: Skipped 1 previous similar message [ 7899.487910] LustreError: 244307:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (0000000079f1a3e1) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7901.610810] Lustre: server umount lustre-MDT0000 complete [ 7902.693441] LustreError: 239619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7902.707822] LustreError: 239619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7905.853138] LustreError: 240358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192793 with bad export cookie 9928574544608086186 [ 7905.860179] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7905.870917] LustreError: 240358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7906.525047] Lustre: server umount lustre-MDT0001 complete [ 7912.569692] Lustre: server umount lustre-OST0000 complete [ 7926.459607] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:06:50 (1784192810) [ 7938.644201] Lustre: server umount lustre-OST0001 complete [ 7946.833777] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 7959.185831] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8001.944480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8002.164457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8002.171203] Lustre: Skipped 2 previous similar messages [ 8002.186471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8002.243290] Lustre: lustre-MDT0000: new disk, initializing [ 8002.326159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8002.350875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8007.185084] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8017.923432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8018.033568] Lustre: 247929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8018.084631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8018.090989] Lustre: Skipped 1 previous similar message [ 8018.183351] Lustre: lustre-MDT0001: new disk, initializing [ 8018.288932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8018.327130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8018.343540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8022.429616] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8031.467883] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8037.319540] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8044.794342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8045.263390] Lustre: lustre-OST0000: new disk, initializing [ 8045.275255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8045.298798] Lustre: 249896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8045.422468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8051.268227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8051.296881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8051.426023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8051.989741] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8062.440514] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8064.907305] LustreError: 247936:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8073.606052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8073.799456] Lustre: lustre-OST0001: new disk, initializing [ 8073.804155] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8073.811443] Lustre: 251012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8074.911147] LustreError: 247936:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8074.933029] LustreError: 247934:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8079.974603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8079.992405] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8080.096323] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8080.403645] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8085.015520] LustreError: 247934:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8090.533119] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8092.280828] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8095.722317] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8095.722878] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8095.740770] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8095.746571] Lustre: Skipped 2 previous similar messages [ 8095.770676] Lustre: Skipped 4 previous similar messages [ 8100.325457] Lustre: server umount lustre-OST0001 complete [ 8100.843319] LustreError: 250200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8100.858262] LustreError: 250200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8118.240728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8118.240962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8118.250449] LustreError: Skipped 2 previous similar messages [ 8118.253521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8118.268665] Lustre: Skipped 1 previous similar message [ 8118.281151] Lustre: Skipped 2 previous similar messages [ 8121.320814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8121.335491] Lustre: Skipped 1 previous similar message [ 8124.063416] Lustre: server umount lustre-OST0000 complete [ 8131.041344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8131.053094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.065546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8133.970412] Lustre: server umount lustre-MDT0000 complete [ 8137.682022] LustreError: 247919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193025 with bad export cookie 9928574544608088790 [ 8137.687299] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8137.693926] LustreError: 247919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8144.525284] Lustre: server umount lustre-MDT0001 complete [ 8155.706952] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8158.834259] Key type lgssc unregistered [ 8159.206747] LNet: 253042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8159.220617] LNetError: 253042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8160.301402] LNet: Removed LNI 192.168.204.120@tcp [ 8161.207191] Key type .llcrypt unregistered [ 8161.213247] Key type ._llcrypt unregistered [ 8204.428136] Key type ._llcrypt registered [ 8204.431902] Key type .llcrypt registered [ 8204.582555] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8206.075466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8206.142373] alg: No test for adler32 (adler32-zlib) [ 8207.178147] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 8207.523271] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8209.231235] Key type lgssc registered [ 8210.394583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8219.164314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8219.174704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8220.364231] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8220.389247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8220.541695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.793914] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8234.977310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.109237] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8235.188389] Lustre: 255251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8235.493650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8240.655571] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8248.372712] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8254.467595] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8262.201132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8262.406715] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8262.595069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.614902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8276.167889] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8291.565254] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8353.250606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8353.264272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.295499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8353.771634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8353.782468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.796590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8356.876527] Lustre: server umount lustre-OST0000 complete [ 8362.092730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8376.271395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8376.579172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.810590] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8387.011309] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8394.980620] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8394.991108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8394.991398] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8395.010938] Lustre: Skipped 1 previous similar message [ 8425.956125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8425.961093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8425.965687] LustreError: Skipped 1 previous similar message [ 8425.966472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8425.984899] Lustre: Skipped 1 previous similar message [ 8426.006065] Lustre: Skipped 1 previous similar message [ 8431.093546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8431.107823] Lustre: Skipped 1 previous similar message [ 8431.691928] Lustre: server umount lustre-OST0000 complete [ 8440.288719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8440.299551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8440.329111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8443.323496] Lustre: server umount lustre-MDT0000 complete [ 8447.665419] LustreError: 255243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193335 with bad export cookie 16874141863436817169 [ 8447.679919] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.120716] Lustre: server umount lustre-MDT0001 complete [ 8461.859201] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8465.322546] Key type lgssc unregistered [ 8465.762833] LNet: 260670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8465.787669] LNetError: 260670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8466.873614] LNet: Removed LNI 192.168.204.120@tcp [ 8468.174516] Key type .llcrypt unregistered [ 8468.182674] Key type ._llcrypt unregistered [ 8494.496672] Key type ._llcrypt registered [ 8494.502845] Key type .llcrypt registered [ 8494.599590] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8495.965673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8495.991330] alg: No test for adler32 (adler32-zlib) [ 8497.104935] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 8497.387270] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8499.063181] Key type lgssc registered [ 8500.514827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8510.302041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8510.326319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8512.061470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8517.451480] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8522.214411] LustreError: 262091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8522.233306] LustreError: 262091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8527.281683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.330655] LustreError: 262090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.680655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8532.293874] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8540.972323] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8548.390531] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8557.382466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8558.070050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8566.431762] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8575.249157] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8581.687578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8607.212696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8607.213834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8607.230585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8607.251670] Lustre: Skipped 1 previous similar message [ 8607.266846] Lustre: Skipped 1 previous similar message [ 8612.324966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8612.340609] Lustre: Skipped 1 previous similar message [ 8612.834881] Lustre: server umount lustre-OST0000 complete [ 8617.441150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8617.455978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8617.465455] Lustre: Skipped 1 previous similar message [ 8622.570861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8622.581622] Lustre: Skipped 1 previous similar message [ 8623.770975] Lustre: server umount lustre-MDT0000 complete [ 8627.683121] LustreError: 263399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8627.710718] LustreError: 263399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8628.761085] LustreError: 262072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193516 with bad export cookie 1281977438301588698 [ 8628.770583] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8629.252733] Lustre: server umount lustre-MDT0001 complete [ 8643.986729] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8647.195693] Key type lgssc unregistered [ 8647.818680] LNet: 266401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8647.837865] LNetError: 266401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8647.871903] LNet: Removed LNI 192.168.204.120@tcp [ 8649.099292] Key type .llcrypt unregistered [ 8649.104335] Key type ._llcrypt unregistered [ 8670.781656] Key type ._llcrypt registered [ 8670.785354] Key type .llcrypt registered [ 8670.943656] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8672.408099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8672.477948] alg: No test for adler32 (adler32-zlib) [ 8673.754506] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 8674.232547] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8676.047359] Key type lgssc registered [ 8677.546367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8686.853799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8686.869875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.439307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8692.921398] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8698.850171] LustreError: 267624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8698.874400] LustreError: 267624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8702.288800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.706634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8707.879011] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8716.215485] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8722.853074] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8730.191733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8730.544218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8733.623987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8737.663744] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8747.629612] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8763.361619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8763.367624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8763.379944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8768.482283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8768.495496] Lustre: Skipped 2 previous similar messages [ 8773.600459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8773.606813] Lustre: Skipped 1 previous similar message [ 8777.183231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8777.326723] Lustre: server umount lustre-OST0000 complete [ 8783.843995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8783.849447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8783.862264] Lustre: Skipped 2 previous similar messages [ 8783.878618] Lustre: Skipped 1 previous similar message [ 8788.524954] Lustre: server umount lustre-MDT0000 complete [ 8788.977221] LustreError: 267620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.999452] LustreError: 267620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8793.095696] LustreError: 268361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193680 with bad export cookie 12869158684826973079 [ 8793.111154] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8793.120519] LustreError: 268361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8793.654393] Lustre: server umount lustre-MDT0001 complete [ 8806.162914] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8809.372607] Key type lgssc unregistered [ 8809.692850] LNet: 271306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8809.718186] LNetError: 271306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8810.791278] LNet: Removed LNI 192.168.204.120@tcp [ 8811.677878] Key type .llcrypt unregistered [ 8811.680534] Key type ._llcrypt unregistered [ 8841.342470] Key type ._llcrypt registered [ 8841.345377] Key type .llcrypt registered [ 8841.447825] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8842.851883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8842.885124] alg: No test for adler32 (adler32-zlib) [ 8844.111659] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 8844.432477] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8846.127487] Key type lgssc registered [ 8847.406510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8856.175343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8856.205634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.756394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.176982] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8867.304661] LustreError: 272727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8867.326110] LustreError: 272727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8871.990171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.566554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8878.865136] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8889.008140] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8895.777464] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8903.006243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8903.338516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.853849] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8920.361257] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8927.949584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8964.066631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8964.067107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8964.087692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8964.091702] Lustre: Skipped 1 previous similar message [ 8965.667583] Lustre: server umount lustre-OST0000 complete [ 8974.305856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.324535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8974.327278] Lustre: Skipped 1 previous similar message [ 8976.420551] Lustre: server umount lustre-MDT0000 complete [ 8979.426277] LustreError: 272727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8979.464456] LustreError: 272727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8980.799513] LustreError: 272705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193868 with bad export cookie 14800249360335584575 [ 8980.802084] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8980.810573] LustreError: 272705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8981.415548] Lustre: server umount lustre-MDT0001 complete [ 8995.113435] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8998.912891] Key type lgssc unregistered [ 8999.385659] LNet: 277056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8999.393930] LNetError: 277056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8999.410752] LNet: Removed LNI 192.168.204.120@tcp [ 9000.686657] Key type .llcrypt unregistered [ 9000.689793] Key type ._llcrypt unregistered [ 9021.157807] Key type ._llcrypt registered [ 9021.160182] Key type .llcrypt registered [ 9021.274610] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9023.022830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9023.103596] alg: No test for adler32 (adler32-zlib) [ 9024.230694] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 9024.653768] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9026.415826] Key type lgssc registered [ 9028.217348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9036.762970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9036.787081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.381990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.952314] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9048.547599] LustreError: 278294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9048.570074] LustreError: 278294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9053.673729] LustreError: 278296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9054.619805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9055.081740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9060.508245] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9068.563791] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9073.654681] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9079.602821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9079.718341] LustreError: 280141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9080.006511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.144386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9086.908115] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9094.463458] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9109.991436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9110.001094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9110.027472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9110.036570] Lustre: Skipped 1 previous similar message [ 9112.051587] Lustre: server umount lustre-OST0000 complete [ 9116.641672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9116.657491] LustreError: Skipped 1 previous similar message [ 9116.667089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9116.677772] Lustre: Skipped 1 previous similar message [ 9116.682090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9120.224040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9120.227471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9122.444486] Lustre: server umount lustre-MDT0000 complete [ 9125.355878] LustreError: 278281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9125.393901] LustreError: 278281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9126.496793] LustreError: 278259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194014 with bad export cookie 12713600585388564028 [ 9126.502415] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9126.512218] LustreError: 278259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9127.071267] Lustre: server umount lustre-MDT0001 complete [ 9140.263851] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9143.869409] Key type lgssc unregistered [ 9144.376868] LNet: 281997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9144.394309] LNetError: 281997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9144.412799] LNet: Removed LNI 192.168.204.120@tcp [ 9145.506892] Key type .llcrypt unregistered [ 9145.513923] Key type ._llcrypt unregistered [ 9158.849867] Key type ._llcrypt registered [ 9158.853736] Key type .llcrypt registered [ 9158.974399] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9176.991900] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9193.442559] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9194.793174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9194.907257] alg: No test for adler32 (adler32-zlib) [ 9195.974155] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 9196.228883] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9197.944428] Key type lgssc registered [ 9198.950579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9237.586403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9237.621166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.040678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9239.117087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9239.321883] Lustre: lustre-MDT0000: new disk, initializing [ 9239.433683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.450638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9244.922759] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9257.035460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9257.123555] Lustre: 285554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9257.179424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9257.183249] Lustre: Skipped 1 previous similar message [ 9257.268575] Lustre: lustre-MDT0001: new disk, initializing [ 9257.356354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9257.397693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9257.412822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9262.809419] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9274.361695] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9280.597704] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9288.872184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9289.369753] Lustre: lustre-OST0000: new disk, initializing [ 9289.379190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9289.385311] Lustre: 287520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9289.470629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.263414] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9299.028454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9299.039523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9299.197161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9309.349719] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9314.400061] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9314.651476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9320.263661] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9320.510383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9325.025545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.029494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9325.068978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9328.989082] Lustre: server umount lustre-OST0000 complete [ 9334.241113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9334.248458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9334.257906] Lustre: Skipped 1 previous similar message [ 9334.262200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9334.266803] Lustre: Skipped 1 previous similar message [ 9335.265150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9335.266490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.552605] Lustre: server umount lustre-MDT0000 complete [ 9343.530563] LustreError: 285544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194231 with bad export cookie 9501377605983167092 [ 9343.533030] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9343.548080] LustreError: 285544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9343.817097] Lustre: server umount lustre-MDT0001 complete [ 9353.122027] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:30:38 (1784194238) [ 9360.254037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.853102] LustreError: 289934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9360.879972] LustreError: 289934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9360.954947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.652770] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9365.988625] LustreError: 289935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.119275] LustreError: 289934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9375.050303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.501784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9380.307613] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9387.953232] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9394.388112] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9402.321247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9402.614181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9409.959602] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9418.336722] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9436.657018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9436.657416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9436.662340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9436.695362] Lustre: Skipped 1 previous similar message [ 9436.737362] Lustre: Skipped 2 previous similar messages [ 9439.219427] Lustre: server umount lustre-OST0000 complete [ 9446.884687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9446.887380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9446.913836] Lustre: Skipped 1 previous similar message [ 9446.924950] Lustre: Skipped 1 previous similar message [ 9455.009165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9455.012899] Lustre: Skipped 3 previous similar messages [ 9458.143128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9458.380493] Lustre: server umount lustre-MDT0000 complete [ 9462.248883] LustreError: 289934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.276118] LustreError: 289934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9462.635249] LustreError: 290668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194350 with bad export cookie 9501377605983167974 [ 9462.645650] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9462.655020] LustreError: 290668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9463.017644] Lustre: server umount lustre-MDT0001 complete [ 9474.499495] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9477.428882] Key type lgssc unregistered [ 9477.715743] LNet: 293494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9477.721358] LNetError: 293494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9477.763622] LNet: Removed LNI 192.168.204.120@tcp [ 9478.651224] Key type .llcrypt unregistered [ 9478.655328] Key type ._llcrypt unregistered [ 9506.653054] Key type ._llcrypt registered [ 9506.658840] Key type .llcrypt registered [ 9506.790606] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9523.991944] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9525.236031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9525.422856] alg: No test for adler32 (adler32-zlib) [ 9526.656509] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [ 9526.918136] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9528.559168] Key type lgssc registered [ 9529.935492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9584.492790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9584.526561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.795823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9585.816592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9585.862122] Lustre: lustre-MDT0000: new disk, initializing [ 9585.923087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9585.940542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9590.985815] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9602.904723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.011263] Lustre: 297326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9603.074573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9603.081392] Lustre: Skipped 1 previous similar message [ 9603.142170] Lustre: lustre-MDT0001: new disk, initializing [ 9603.215695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9603.258774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9603.270198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9607.687626] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9617.438744] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9623.307087] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9629.953111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9630.296289] Lustre: lustre-OST2710: new disk, initializing [ 9630.305334] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9630.316369] Lustre: 299295:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9630.430434] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9636.508951] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9639.010926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9639.027242] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9639.130509] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9646.283310] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9654.040699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9654.246694] Lustre: lustre-OST03e8: new disk, initializing [ 9654.250726] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9654.262655] Lustre: 300357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9654.376547] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9660.012728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9660.023611] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9660.134563] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9661.202806] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9668.775661] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9678.212306] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9678.525464] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9684.096558] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9684.407732] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9689.596740] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9689.926795] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9695.218318] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9695.500968] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9700.837938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9700.848641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9700.866459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9702.865196] Lustre: server umount lustre-MDT0000 complete [ 9704.428514] LustreError: 297338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9704.429944] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.442924] LustreError: 297338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9706.568562] LustreError: 297318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194594 with bad export cookie 13088901016164848690 [ 9706.577077] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9706.592793] LustreError: 297318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9706.879261] Lustre: server umount lustre-MDT0001 complete [ 9720.959578] Lustre: server umount lustre-OST2710 complete [ 9735.485876] Lustre: server umount lustre-OST03e8 complete [ 9754.942847] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9766.461405] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9805.179277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9805.555194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9805.625640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9805.741557] Lustre: lustre-MDT0000: new disk, initializing [ 9805.859564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9805.885539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9809.880211] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9820.206822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9820.335575] Lustre: 305611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9820.379725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9820.382761] Lustre: Skipped 1 previous similar message [ 9820.437730] Lustre: lustre-MDT0001: new disk, initializing [ 9820.504124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9820.525560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9820.533237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9823.715573] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9833.123284] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9839.435341] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9846.599203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9846.929647] Lustre: lustre-OST0000: new disk, initializing [ 9846.934197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9846.954361] Lustre: 307579:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9847.169345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9853.214406] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9854.037076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9854.041201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9854.137732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9863.359757] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9868.079033] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9868.259320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9872.564669] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9872.795585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9874.915605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9874.916358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9874.919349] LustreError: Skipped 1 previous similar message [ 9874.920921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9874.929378] Lustre: Skipped 3 previous similar messages [ 9880.045573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9880.049780] Lustre: Skipped 2 previous similar messages [ 9885.158712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9885.171845] Lustre: Skipped 1 previous similar message [ 9889.247173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9889.464941] Lustre: server umount lustre-OST0000 complete [ 9895.393126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9895.397842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9895.424192] Lustre: Skipped 1 previous similar message [ 9905.635475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9905.638640] Lustre: Skipped 4 previous similar messages [ 9908.191463] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9908.380907] Lustre: server umount lustre-MDT0000 complete [ 9910.755535] LustreError: 305617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9910.777342] LustreError: 305617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9912.121246] LustreError: 305601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194799 with bad export cookie 13088901016164850370 [ 9912.128566] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9912.132468] LustreError: 305601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9912.570488] Lustre: server umount lustre-MDT0001 complete [ 9922.040833] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:40:07 (1784194807) [ 9923.963089] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9925.630521] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:40:10 (1784194810) [ 9929.228190] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9957.330301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.544995] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9957.572879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9957.769937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9963.482267] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9973.875401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.065389] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9974.134562] Lustre: 311436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9980.852850] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10003.762900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10004.072660] Lustre: Denying initial registration attempt from nid 192.168.204.120@tcp, specified as failover [10004.087175] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10004.102650] LustreError: 312890:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10004.114687] LustreError: 312890:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10004.121773] LustreError: 312890:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10004.179359] Lustre: server umount lustre-OST0000 complete [10004.190364] LustreError: 312890:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [10011.621520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10011.625048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10011.651972] Lustre: Skipped 1 previous similar message [10011.677973] Lustre: Skipped 1 previous similar message [10017.586515] Lustre: server umount lustre-MDT0000 complete [10021.270847] LustreError: 313261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194909 with bad export cookie 13088901016164851245 [10021.283836] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10021.290944] LustreError: 313261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10021.740712] Lustre: server umount lustre-MDT0001 complete [10034.563839] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10038.110675] Key type lgssc unregistered [10038.555664] LNet: 314013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10038.560061] LNetError: 314013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10038.590367] LNet: Removed LNI 192.168.204.120@tcp [10039.736356] Key type .llcrypt unregistered [10039.746412] Key type ._llcrypt unregistered [10053.529648] Key type ._llcrypt registered [10053.532962] Key type .llcrypt registered [10053.632801] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10054.846483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10054.873749] alg: No test for adler32 (adler32-zlib) [10056.002128] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [10056.236904] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10057.983168] Key type lgssc registered [10059.168673] Lustre: Echo OBD driver; http://www.lustre.org/ [10106.513207] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10114.776965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10114.797991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.056971] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10116.087529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10116.361878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10120.444381] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10128.680828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10128.752342] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10128.774966] Lustre: 316777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10129.032482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10133.132837] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10140.048417] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10145.780566] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10165.036726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10165.291697] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10165.546154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10171.991548] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10180.004249] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10185.703475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10185.729109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10185.759571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10186.208455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10188.908653] Lustre: server umount lustre-OST0000 complete [10195.424916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10195.435606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10195.450788] Lustre: Skipped 1 previous similar message [10195.459873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10195.471205] Lustre: Skipped 2 previous similar messages [10196.451653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10196.466888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10196.470942] Lustre: Skipped 1 previous similar message [10200.037755] Lustre: server umount lustre-MDT0000 complete [10205.423602] LustreError: 316770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195093 with bad export cookie 10978806115078614877 [10205.430144] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10205.442543] LustreError: 316770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10206.041899] Lustre: server umount lustre-MDT0001 complete [10216.866976] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10219.489219] Key type lgssc unregistered [10219.761446] LNet: 320355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10219.769679] LNetError: 320355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10219.812560] LNet: Removed LNI 192.168.204.120@tcp [10220.915993] Key type .llcrypt unregistered [10220.920190] Key type ._llcrypt unregistered [10245.612578] Key type ._llcrypt registered [10245.615281] Key type .llcrypt registered [10245.730185] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10246.536737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10246.659271] alg: No test for adler32 (adler32-zlib) [10247.654843] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [10247.905027] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10249.567285] Key type lgssc registered [10250.684739] Lustre: Echo OBD driver; http://www.lustre.org/ [10257.905480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10257.921431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.354185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10263.523285] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10269.674381] LustreError: 321773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10269.684576] LustreError: 321773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10271.895716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10272.287363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10276.470789] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10284.032516] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10289.710804] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10298.238864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10298.640481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10305.209792] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10312.802749] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10327.524734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.528178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10327.550407] Lustre: Skipped 2 previous similar messages [10327.559025] Lustre: Skipped 1 previous similar message [10330.054256] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [10330.065463] Lustre: Skipped 1 previous similar message [10332.646036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10332.887662] Lustre: server umount lustre-MDT0000 complete [10335.143923] LustreError: 321768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10337.408386] LustreError: 321753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195225 with bad export cookie 9529874181230296897 [10337.411380] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10337.418086] LustreError: 321753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10337.767376] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10337.770344] LustreError: 321770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10337.794033] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10337.824626] Lustre: Skipped 2 previous similar messages [10340.255332] LustreError: 321769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.884031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10342.887623] Lustre: Skipped 1 previous similar message [10343.854541] Lustre: server umount lustre-MDT0001 complete [10351.695995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10361.725344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.338272] LustreError: 321082:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000092e957d7 x1870864197924352/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10362.829568] LustreError: 323605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10362.866567] LustreError: 323605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10362.966430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10363.039285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10368.476598] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10373.091045] LustreError: 325383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.099372] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10373.130159] LustreError: 325383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10377.492588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10377.900245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10382.504812] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10383.348255] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10390.490466] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10392.627974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10398.609805] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10400.162493] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10408.933648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.933800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10408.941188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10408.951629] Lustre: Skipped 1 previous similar message [10411.190223] Lustre: server umount lustre-OST0000 complete [10418.656495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10418.665646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10421.356133] Lustre: server umount lustre-MDT0000 complete [10424.288758] LustreError: 325384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.313507] LustreError: 325384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10425.404919] LustreError: 321753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195313 with bad export cookie 9529874181230304730 [10425.417761] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10425.440203] LustreError: 321753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10425.855650] Lustre: server umount lustre-MDT0001 complete [10438.233261] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10441.362460] Key type lgssc unregistered [10441.694293] LNet: 328059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10441.708608] LNetError: 328059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10441.730556] LNet: Removed LNI 192.168.204.120@tcp [10442.772796] Key type .llcrypt unregistered [10442.775072] Key type ._llcrypt unregistered [10469.636599] Key type ._llcrypt registered [10469.638790] Key type .llcrypt registered [10469.770504] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10471.247771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10471.291632] alg: No test for adler32 (adler32-zlib) [10472.473360] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [10472.724661] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10474.375207] Key type lgssc registered [10475.458216] Lustre: Echo OBD driver; http://www.lustre.org/ [10483.661493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10483.696048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.087284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10489.301219] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10494.445310] LustreError: 329478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.475343] LustreError: 329478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10498.468938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10498.898313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10503.354271] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10511.937707] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10518.421716] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10521.944527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10521.949815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10521.975121] Lustre: Skipped 1 previous similar message [10521.987664] Lustre: Skipped 1 previous similar message [10524.002545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10524.016738] Lustre: Skipped 1 previous similar message [10528.263874] Lustre: server umount lustre-MDT0000 complete [10529.249143] LustreError: 329804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10529.277547] LustreError: 329804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10532.294653] LustreError: 329458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195420 with bad export cookie 17435002308109257023 [10532.306619] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10532.313319] LustreError: 329458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10532.621576] Lustre: server umount lustre-MDT0001 complete [10548.006966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10548.379466] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10548.419687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10548.632396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10553.324666] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10562.649172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10562.729723] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10562.760217] Lustre: 332066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10563.077562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10568.321037] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10577.555595] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10585.213185] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10595.952912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10596.152297] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10596.159968] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10596.168341] LustreError: 333949:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10596.178718] LustreError: 333949:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10596.184202] LustreError: 333949:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10596.279768] Lustre: server umount lustre-OST0000 complete [10596.284669] LustreError: 333949:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10606.186024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10606.413121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10606.627051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10611.828427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10612.613856] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10620.985995] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10629.249481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10629.314320] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10629.371864] Lustre: lustre-OST0001: new disk, initializing [10629.375927] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10629.381900] Lustre: 335493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10629.483903] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10636.250549] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10636.943793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10636.950453] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10637.066862] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10644.824213] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10652.641355] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10652.652324] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10652.656840] LustreError: Skipped 1 previous similar message [10652.682819] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10652.689504] Lustre: Skipped 1 previous similar message [10654.509707] Lustre: server umount lustre-OST0001 complete [10657.778168] LustreError: 334418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10657.807420] LustreError: 334418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10662.884235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10662.904053] LustreError: 334417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10662.905562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.905568] Lustre: Skipped 1 previous similar message [10662.915285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10662.933078] LustreError: 334417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10662.946620] Lustre: Skipped 1 previous similar message [10664.914192] Lustre: server umount lustre-OST0000 complete [10670.566450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10670.587686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10670.606638] Lustre: Skipped 1 previous similar message [10670.617471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10682.337286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10682.355703] Lustre: Skipped 5 previous similar messages [10683.359422] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10683.579981] Lustre: server umount lustre-MDT0000 complete [10687.458729] LustreError: 332077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10687.475913] LustreError: 332077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10688.070981] LustreError: 332057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195575 with bad export cookie 17435002308109257576 [10688.072635] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10688.079439] LustreError: 332057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10688.566508] Lustre: server umount lustre-MDT0001 complete [10702.926533] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10706.978185] Key type lgssc unregistered [10707.515789] LNet: 337454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10707.529370] LNetError: 337454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10707.558707] LNet: Removed LNI 192.168.204.120@tcp [10708.700219] Key type .llcrypt unregistered [10708.706475] Key type ._llcrypt unregistered [10787.016490] Key type ._llcrypt registered [10787.018554] Key type .llcrypt registered [10787.158358] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [10803.434187] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10805.010145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10805.114044] alg: No test for adler32 (adler32-zlib) [10806.147730] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [10806.472937] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10808.159182] Key type lgssc registered [10809.535573] Lustre: Echo OBD driver; http://www.lustre.org/ [10847.337505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10847.380088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10848.680841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10848.713537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10848.833923] Lustre: lustre-MDT0000: new disk, initializing [10848.956546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10848.995380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10853.775784] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10867.014482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10867.310278] Lustre: 342395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10867.415053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10867.428647] Lustre: Skipped 1 previous similar message [10867.563890] Lustre: lustre-MDT0001: new disk, initializing [10867.758562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10867.816525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10867.837199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10873.335152] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10884.138538] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10890.071840] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10897.981958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10898.435488] Lustre: lustre-OST0000: new disk, initializing [10898.438834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10898.463168] Lustre: 344364:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10898.623206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10905.746086] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10906.685846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10906.692063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10906.884034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10915.951969] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10921.475370] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10921.732646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10926.653905] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10926.976938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10932.709094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10932.709317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10932.711957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10932.715337] LustreError: Skipped 1 previous similar message [10932.747238] Lustre: Skipped 1 previous similar message [10935.254945] Lustre: server umount lustre-OST0000 complete [10939.363087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10939.368590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.381632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10939.387042] Lustre: Skipped 1 previous similar message [10942.947810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.951115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10942.981416] Lustre: Skipped 1 previous similar message [10948.071422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10948.077523] Lustre: Skipped 1 previous similar message [10953.183126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10953.190445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10953.203529] Lustre: Skipped 1 previous similar message [10953.380667] Lustre: server umount lustre-MDT0000 complete [10956.845363] LustreError: 344374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195844 with bad export cookie 9646659317467889489 [10956.847508] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10956.868443] LustreError: 344374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10957.218642] Lustre: server umount lustre-MDT0001 complete [10965.653784] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:57:30 (1784195850) [10973.722651] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:57:38 (1784195858) [10983.114922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.433304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10988.519072] LustreError: 347123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10988.554301] LustreError: 347123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10988.632561] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10993.632487] LustreError: 347122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.695787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.764114] LustreError: 347123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.088893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11003.362235] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11010.148582] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11015.945626] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11022.641439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.988961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11028.340369] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11035.705539] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11044.180351] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11046.745640] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11048.798506] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11050.744396] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11052.977112] Lustre: Failing over lustre-MDT0000 [11053.466243] Lustre: server umount lustre-MDT0000 complete [11054.003133] LustreError: 347845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11055.073176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11055.084768] Lustre: Skipped 2 previous similar messages [11059.114841] LustreError: 347117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.127289] LustreError: 347117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11069.357493] LustreError: 347118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11069.369128] LustreError: 347118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11070.431517] Lustre: 340622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784195942/real 1784195942] req@00000000bd866ecc x1870864784103424/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784195958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11070.474860] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11072.655119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.674338] LustreError: 340619:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e710e451 x1870864784108800/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11080.947531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11081.000546] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11084.705038] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11085.147415] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11086.321996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11086.353037] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11086.380404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11094.992661] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11096.908647] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11100.852595] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11102.867108] Lustre: Failing over lustre-MDT0000 [11103.147260] Lustre: server umount lustre-MDT0000 complete [11105.229937] LustreError: 347118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11105.251702] LustreError: 347118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [11106.793925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11106.800208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.827622] Lustre: Skipped 2 previous similar messages [11121.950936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.054423] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11122.146464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [11122.156480] Lustre: Skipped 2 previous similar messages [11122.279639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11122.330457] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11125.672563] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11127.123603] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11127.793797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11127.800764] Lustre: Skipped 2 previous similar messages [11127.835586] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11127.891264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11136.874492] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11138.282589] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11148.259894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11148.262786] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11148.269929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11148.297556] Lustre: Skipped 2 previous similar messages [11151.498768] Lustre: server umount lustre-MDT0000 complete [11153.379637] LustreError: 347122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.405598] LustreError: 347122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [11155.202925] LustreError: 351090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196042 with bad export cookie 9646659317467892205 [11155.211619] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11155.216760] LustreError: 351090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11155.624303] Lustre: server umount lustre-MDT0001 complete [11170.176328] Lustre: server umount lustre-OST0000 complete [11185.978938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.595160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11191.552179] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11200.107035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.564414] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11211.086952] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11216.023195] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11221.758197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11228.746902] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11232.237720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11235.216838] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11241.441937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11241.451788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11241.473399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11241.482855] Lustre: Skipped 2 previous similar messages [11247.350570] Lustre: server umount lustre-OST0000 complete [11251.681147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11251.701120] LustreError: Skipped 1 previous similar message [11257.485712] Lustre: server umount lustre-MDT0000 complete [11257.830547] LustreError: 353475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11257.862317] LustreError: 353475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11261.892224] LustreError: 353457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196149 with bad export cookie 9646659317467892947 [11261.903823] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11261.906729] LustreError: 353457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11262.237734] Lustre: server umount lustre-MDT0001 complete [11274.928461] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11277.865666] Key type lgssc unregistered [11278.129838] LNet: 357007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11278.144504] LNetError: 357007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11278.164690] LNet: Removed LNI 192.168.204.120@tcp [11278.990989] Key type .llcrypt unregistered [11278.992648] Key type ._llcrypt unregistered [11305.878173] Key type ._llcrypt registered [11305.883439] Key type .llcrypt registered [11305.991375] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11307.120631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11307.141785] alg: No test for adler32 (adler32-zlib) [11308.177773] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [11308.397273] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11310.063852] Key type lgssc registered [11311.214743] Lustre: Echo OBD driver; http://www.lustre.org/ [11318.989576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11319.003425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11320.698581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11326.349772] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11331.050093] LustreError: 358525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.075810] LustreError: 358525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11335.645515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.060523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11341.642670] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11349.879211] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11357.262558] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11365.179777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11365.638123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11366.684710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11372.661941] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11379.946349] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11391.473710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.482230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11391.505091] Lustre: Skipped 1 previous similar message [11391.527084] Lustre: Skipped 2 previous similar messages [11391.800432] Lustre: server umount lustre-MDT0000 complete [11395.510738] LustreError: 358506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196283 with bad export cookie 9095662530781121050 [11395.528461] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11395.531621] LustreError: 358506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11396.000405] Lustre: server umount lustre-MDT0001 complete [11409.512317] Lustre: server umount lustre-OST0000 complete [11424.380285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11424.751107] LustreError: 362239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11424.831088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11429.799149] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11429.866687] LustreError: 362240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11434.981042] LustreError: 362239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.103197] LustreError: 362240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11441.592288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11442.129523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11447.719121] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11456.822152] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11463.423780] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11470.920169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11471.385265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11478.624453] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11487.024690] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11494.962968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11513.832536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11513.848072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.861701] Lustre: Skipped 1 previous similar message [11513.879373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11515.362500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11515.366840] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11516.960801] Lustre: server umount lustre-MDT0000 complete [11520.490074] LustreError: 364383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11520.515219] LustreError: 364383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11520.638823] LustreError: 362220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196408 with bad export cookie 9095662530781122163 [11520.639876] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11520.650641] LustreError: 362220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11521.042640] Lustre: server umount lustre-MDT0001 complete [11535.989391] Lustre: server umount lustre-OST0000 complete [11552.315471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11552.893367] LustreError: 366175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11553.009642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11558.142293] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11568.863175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11574.205475] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11583.077320] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11588.807177] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11595.821904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11596.198555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11596.204987] Lustre: Skipped 1 previous similar message [11598.258528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11602.391598] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11609.850202] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11615.721755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11615.728948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11615.746640] Lustre: Skipped 1 previous similar message [11615.762660] Lustre: Skipped 3 previous similar messages [11616.888711] Lustre: server umount lustre-OST0000 complete [11620.842413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11620.844172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11620.845550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11620.855037] Lustre: Skipped 1 previous similar message [11626.797941] Lustre: server umount lustre-MDT0000 complete [11630.923354] LustreError: 366156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196518 with bad export cookie 9095662530781123178 [11630.931709] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11630.935924] LustreError: 366156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11631.381979] Lustre: server umount lustre-MDT0001 complete [11643.814811] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11646.857806] Key type lgssc unregistered [11647.120912] LNet: 369710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11647.126899] LNetError: 369710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11647.147941] LNet: Removed LNI 192.168.204.120@tcp [11648.059354] Key type .llcrypt unregistered [11648.065667] Key type ._llcrypt unregistered [11677.920153] Key type ._llcrypt registered [11677.924986] Key type .llcrypt registered [11678.056435] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11679.903173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11679.938995] alg: No test for adler32 (adler32-zlib) [11681.092453] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [11681.471428] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11683.263169] Key type lgssc registered [11684.520724] Lustre: Echo OBD driver; http://www.lustre.org/ [11692.866145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11692.885317] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.902417] LustreError: 371152:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11692.938382] LustreError: 371152:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11692.949740] LustreError: 371152:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11692.955236] LustreError: 371152:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11692.966985] LustreError: 371152:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11701.411209] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11701.416247] LustreError: 371549:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11701.432552] LustreError: 371549:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11701.440151] LustreError: 371549:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11701.447266] LustreError: 371549:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11701.454834] LustreError: 371549:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11723.111496] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11725.889363] Key type lgssc unregistered [11726.220779] LNet: 372554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11726.226774] LNetError: 372554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11726.242392] LNet: Removed LNI 192.168.204.120@tcp [11727.127819] Key type .llcrypt unregistered [11727.136567] Key type ._llcrypt unregistered [11745.987605] Key type ._llcrypt registered [11745.991728] Key type .llcrypt registered [11746.114451] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [11764.562262] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11766.148043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11766.174703] alg: No test for adler32 (adler32-zlib) [11767.259570] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [11767.612844] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11769.327578] Key type lgssc registered [11770.716630] Lustre: Echo OBD driver; http://www.lustre.org/ [11812.371212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11812.432710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11813.668868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11813.702877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11813.779969] Lustre: lustre-MDT0000: new disk, initializing [11813.923589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11813.944121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11819.340855] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11831.435476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11831.580879] Lustre: 375716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11831.656474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11831.662500] Lustre: Skipped 1 previous similar message [11831.731984] Lustre: lustre-MDT0001: new disk, initializing [11831.822248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11831.864315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11831.876609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11837.050642] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11847.826629] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11854.701530] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11863.028369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11863.267506] Lustre: lustre-OST0000: new disk, initializing [11863.271434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11863.280203] Lustre: 377687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11863.382387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11870.023742] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11870.294923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11870.309253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11870.450780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11879.910654] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11885.407684] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11885.633984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11890.295714] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11890.508521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11896.295382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11896.298154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11896.302526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11896.304597] LustreError: Skipped 1 previous similar message [11896.344156] Lustre: Skipped 1 previous similar message [11898.966926] Lustre: server umount lustre-OST0000 complete [11903.456175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11903.487991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11903.506129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11903.509404] Lustre: Skipped 1 previous similar message [11906.532609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11906.543522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11906.568908] Lustre: Skipped 1 previous similar message [11909.388972] Lustre: server umount lustre-MDT0000 complete [11913.093273] LustreError: 375709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196800 with bad export cookie 6925663863198984590 [11913.098068] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11913.109020] LustreError: 375709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11913.422479] Lustre: server umount lustre-MDT0001 complete [11921.359579] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11923.228247] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:13:28 (1784196808) [11929.917309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.353437] LustreError: 380145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11930.377836] LustreError: 380145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11930.457744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11935.024911] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11935.712740] LustreError: 380146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11940.836893] LustreError: 380145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.702451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11943.005891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11947.086330] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11955.501890] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11961.268443] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11968.577097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11968.887660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11975.192549] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11983.720932] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11991.953251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11992.188987] Lustre: lustre-OST0001: new disk, initializing [11992.194607] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11992.201275] Lustre: 382980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11992.319427] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11999.557894] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11999.920278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11999.934995] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12000.126603] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12010.505573] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12015.584062] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12015.588893] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12015.591881] LustreError: Skipped 1 previous similar message [12015.594901] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12015.611220] Lustre: Skipped 1 previous similar message [12020.605615] Lustre: server umount lustre-OST0001 complete [12020.712336] LustreError: 381970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.744245] LustreError: 381970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12025.831417] LustreError: 381969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12025.856592] LustreError: 381969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12036.069037] LustreError: 381970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.091880] LustreError: 381970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12038.624911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12038.644311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12038.661536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12038.677358] Lustre: Skipped 2 previous similar messages [12044.046797] Lustre: server umount lustre-OST0000 complete [12050.403156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12050.418356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12050.430347] Lustre: Skipped 1 previous similar message [12050.450565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12050.463060] Lustre: Skipped 2 previous similar messages [12053.993942] Lustre: server umount lustre-MDT0000 complete [12056.553290] LustreError: 380145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12056.577543] LustreError: 380145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12057.757874] LustreError: 380125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196945 with bad export cookie 6925663863198985472 [12057.769416] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12064.184823] Lustre: server umount lustre-MDT0001 complete [12076.807217] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12079.561657] Key type lgssc unregistered [12079.819458] LNet: 384972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12079.829414] LNetError: 384972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12079.853895] LNet: Removed LNI 192.168.204.120@tcp [12080.626289] Key type .llcrypt unregistered [12080.628432] Key type ._llcrypt unregistered [12128.958560] Key type ._llcrypt registered [12128.960689] Key type .llcrypt registered [12129.042783] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12129.815033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12129.843248] alg: No test for adler32 (adler32-zlib) [12130.892547] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [12131.170216] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12132.839137] Key type lgssc registered [12133.793207] Lustre: Echo OBD driver; http://www.lustre.org/ [12139.895762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12139.914343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12141.111537] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12141.126805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12141.247291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12145.006233] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12151.943850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12152.037493] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12152.081255] Lustre: 387772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12152.357205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12156.751948] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12163.366355] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12168.217603] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12172.768430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12172.773526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12172.779889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12172.792404] Lustre: Skipped 1 previous similar message [12176.263267] Lustre: server umount lustre-MDT0000 complete [12179.633857] LustreError: 387763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197067 with bad export cookie 6887908372655604949 [12179.635671] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12179.642839] LustreError: 387763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12179.929149] Lustre: server umount lustre-MDT0001 complete [12188.688805] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:17:54 (1784197074) [12194.429130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12194.827602] LustreError: 390283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12194.845238] LustreError: 390283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12194.883132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12198.539124] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12199.907094] LustreError: 390284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12205.024453] LustreError: 390283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12206.221113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.531809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12210.334837] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12217.401461] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12223.539405] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12230.339969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12230.535846] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12230.716378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12236.786216] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12244.095613] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12250.702790] Lustre: Permanently deactivating lustre-OST0000 [12250.729729] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12256.784639] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12258.419870] LustreError: 393223:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12258.428706] LustreError: 393223:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12258.436850] LustreError: 393223:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12261.033689] LustreError: 393271:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12261.040417] LustreError: 393271:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12269.097912] Lustre: server umount lustre-OST0000 complete [12271.454908] LustreError: 393521:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12271.459323] LustreError: 393521:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12273.634649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12273.659561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12273.665143] Lustre: Skipped 1 previous similar message [12278.242149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12278.248468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12278.258451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12282.956901] Lustre: server umount lustre-MDT0000 complete [12283.879598] LustreError: 392411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12283.904740] LustreError: 392411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12286.217256] LustreError: 390263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197173 with bad export cookie 6887908372655605530 [12286.217851] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12286.223418] LustreError: 390263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12295.560813] Lustre: server umount lustre-MDT0001 complete [12302.300819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12305.719516] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12307.701538] LustreError: 394678:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12313.283587] Lustre: 395017:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.120@tcp [12315.257801] Lustre: server umount MGS complete [12324.570201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12324.897688] LustreError: 395741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12324.915732] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12324.919486] Lustre: Skipped 1 previous similar message [12324.970853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12328.465548] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12334.974975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12335.158507] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12338.493425] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12344.770536] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12349.339736] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12350.837488] Lustre: Permanently reactivating lustre-OST0000 [12350.854776] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12350.862163] Lustre: Skipped 2 previous similar messages [12380.129214] LustreError: 397600:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12385.232122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12385.484906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12385.492215] Lustre: Skipped 1 previous similar message [12390.844250] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12397.416513] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12405.226853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12405.228990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12405.240145] LustreError: Skipped 1 previous similar message [12405.240957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12405.254247] Lustre: Skipped 1 previous similar message [12405.265081] Lustre: Skipped 2 previous similar messages [12408.636565] Lustre: server umount lustre-OST0000 complete [12415.459301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12415.468357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12415.473068] Lustre: Skipped 1 previous similar message [12418.575709] Lustre: server umount lustre-MDT0000 complete [12420.580543] LustreError: 395741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12420.592045] LustreError: 395741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12421.738529] LustreError: 395722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197309 with bad export cookie 6887908372655606580 [12421.741953] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12421.756395] LustreError: 395722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12422.020624] Lustre: server umount lustre-MDT0001 complete [12430.645585] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12432.572470] Key type lgssc unregistered [12432.804281] LNet: 400247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12432.809465] LNetError: 400247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12432.819742] LNet: Removed LNI 192.168.204.120@tcp [12433.438482] Key type .llcrypt unregistered [12433.441148] Key type ._llcrypt unregistered [12448.338660] Key type ._llcrypt registered [12448.341355] Key type .llcrypt registered [12448.437085] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [12461.684202] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12462.765427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12462.788527] alg: No test for adler32 (adler32-zlib) [12463.807472] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [12464.103699] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12465.783429] Key type lgssc registered [12466.986274] Lustre: Echo OBD driver; http://www.lustre.org/ [12496.554588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12496.577286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12497.816446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12497.856831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12497.925602] Lustre: lustre-MDT0000: new disk, initializing [12497.979107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12497.995589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12502.207722] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12511.480668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12511.610907] Lustre: 403406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12511.659418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12511.664812] Lustre: Skipped 1 previous similar message [12511.735386] Lustre: lustre-MDT0001: new disk, initializing [12511.827351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12511.858016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12511.868880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12515.947853] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12524.408837] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12530.214156] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12536.068429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12536.359525] Lustre: lustre-OST0000: new disk, initializing [12536.363493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12536.368832] Lustre: 405376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12536.446755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12541.934278] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12544.587799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12544.599250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12544.754947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12550.148215] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12554.201317] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12554.400267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12557.498716] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12557.652862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12560.353349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12560.354640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12560.365740] LustreError: Skipped 1 previous similar message [12560.369106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12560.376472] Lustre: Skipped 1 previous similar message [12560.391175] Lustre: Skipped 1 previous similar message [12565.263775] Lustre: server umount lustre-OST0000 complete [12570.592976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12570.595414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12570.607058] Lustre: Skipped 1 previous similar message [12570.615258] Lustre: Skipped 1 previous similar message [12575.713827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12575.724943] Lustre: Skipped 1 previous similar message [12580.832602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12580.840582] Lustre: Skipped 1 previous similar message [12583.391248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12583.531848] Lustre: server umount lustre-MDT0000 complete [12586.272583] LustreError: 403396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197474 with bad export cookie 1346084637592782413 [12586.273944] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12586.286644] LustreError: 403396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12586.615861] Lustre: server umount lustre-MDT0001 complete [12593.509375] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12595.200290] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:24:40 (1784197480) [12601.718089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12602.135280] LustreError: 407835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.154655] LustreError: 407835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12602.217689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12606.028540] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12607.457660] LustreError: 407836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.563101] LustreError: 407835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12613.468304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12613.804862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12617.368266] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12623.522851] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12628.917192] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12635.825417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12636.259244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12642.994439] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12650.172430] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12652.616129] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12655.570446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12662.245963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12662.247936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12662.248685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12662.275097] Lustre: Skipped 1 previous similar message [12667.135614] Lustre: server umount lustre-OST0000 complete [12672.485042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12672.500293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12672.517652] Lustre: Skipped 2 previous similar messages [12676.973134] Lustre: server umount lustre-MDT0000 complete [12677.600865] LustreError: 408564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12677.618879] LustreError: 408564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12680.562701] LustreError: 407815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197568 with bad export cookie 1346084637592783295 [12680.578031] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12680.580284] LustreError: 407815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12680.868879] Lustre: server umount lustre-MDT0001 complete [12690.392830] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12693.025135] Key type lgssc unregistered [12693.270629] LNet: 411561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12693.276631] LNetError: 411561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12693.294086] LNet: Removed LNI 192.168.204.120@tcp [12694.114139] Key type .llcrypt unregistered [12694.116904] Key type ._llcrypt unregistered [12716.645934] Key type ._llcrypt registered [12716.649940] Key type .llcrypt registered [12716.764579] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12717.005436] Key type .llcrypt unregistered [12717.007781] Key type ._llcrypt unregistered [12733.382918] Key type ._llcrypt registered [12733.384742] Key type .llcrypt registered [12733.479884] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12734.581195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12734.604237] alg: No test for adler32 (adler32-zlib) [12735.590385] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [12735.843144] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12737.536054] Key type lgssc registered [12738.446694] Lustre: Echo OBD driver; http://www.lustre.org/ [12745.187877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12745.215599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12746.638993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12750.642977] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12756.966238] LustreError: 413774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12756.979264] LustreError: 413774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12758.465752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12758.766644] LustreError: 413775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12758.802393] LustreError: 413775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12758.815549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12764.130358] LustreError: 413769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12764.147041] LustreError: 413769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12764.492715] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12769.249581] LustreError: 413771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12769.282114] LustreError: 413771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12771.028585] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12771.312945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12776.420543] LustreError: 413774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12776.438244] LustreError: 413774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12777.037405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12777.361335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12780.861794] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12790.885561] Lustre: server umount lustre-OST0000 complete [12796.904761] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12796.911328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12796.926965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12797.931110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12797.934146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12797.953463] Lustre: Skipped 1 previous similar message [12803.041760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12803.055443] Lustre: Skipped 1 previous similar message [12804.143891] Lustre: server umount lustre-MDT0000 complete [12807.130924] LustreError: 413756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197694 with bad export cookie 5717887492918035496 [12807.132686] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12807.136802] LustreError: 413756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12816.494170] Lustre: server umount lustre-MDT0001 complete [12824.978603] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12827.391330] Key type lgssc unregistered [12827.627910] LNet: 416922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12827.634082] LNetError: 416922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12827.645051] LNet: Removed LNI 192.168.204.120@tcp [12828.350774] Key type .llcrypt unregistered [12828.353330] Key type ._llcrypt unregistered [12850.113335] Key type ._llcrypt registered [12850.115923] Key type .llcrypt registered [12850.221542] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12851.021481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12851.135064] alg: No test for adler32 (adler32-zlib) [12852.244531] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [12852.486915] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12854.157142] Key type lgssc registered [12855.438104] Lustre: Echo OBD driver; http://www.lustre.org/ [12863.049558] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12863.064949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12878.560863] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12883.679481] LustreError: 418304:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [12910.369676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12915.894934] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12922.022543] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12928.420751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12928.912340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12928.984449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12932.885837] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12934.114956] LustreError: 418329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12934.135756] LustreError: 418329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12939.237847] LustreError: 418330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12940.364570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12940.639995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12944.134691] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12949.910745] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12953.850411] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12960.226601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12960.231908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12960.246435] Lustre: Skipped 1 previous similar message [12962.078215] Lustre: server umount lustre-OST0000 complete [12965.347234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12965.349595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12965.362937] Lustre: Skipped 1 previous similar message [12965.367432] Lustre: Skipped 1 previous similar message [12970.466541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12970.472920] Lustre: Skipped 2 previous similar messages [12971.473529] Lustre: server umount lustre-MDT0000 complete [12974.878486] LustreError: 418312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197862 with bad export cookie 1988936764558685258 [12974.882803] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12974.888364] LustreError: 418312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12975.125886] Lustre: server umount lustre-MDT0001 complete [12984.377000] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12986.443532] Key type lgssc unregistered [12986.641964] LNet: 421873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12986.648765] LNetError: 421873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12986.664171] LNet: Removed LNI 192.168.204.120@tcp [12987.314618] Key type .llcrypt unregistered [12987.316252] Key type ._llcrypt unregistered [13006.435885] Key type ._llcrypt registered [13006.438176] Key type .llcrypt registered [13006.541695] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13007.182733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13007.259226] alg: No test for adler32 (adler32-zlib) [13008.297873] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [13008.503917] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13010.177505] Key type lgssc registered [13011.018982] Lustre: Echo OBD driver; http://www.lustre.org/ [13016.836360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13016.857431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13018.259850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13021.942941] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13027.811352] LustreError: 423270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13027.832392] LustreError: 423270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13028.864092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13029.202776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13033.068647] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13039.605477] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13044.018692] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13049.344686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13049.633856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13054.276775] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13060.346643] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13062.968468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13068.268793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13068.273067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13068.285488] Lustre: Skipped 2 previous similar messages [13068.301128] Lustre: Skipped 2 previous similar messages [13071.296569] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [13073.376942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13073.389398] Lustre: Skipped 2 previous similar messages [13074.303095] Lustre: server umount lustre-MDT0000 complete [13076.308487] LustreError: 423250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.20@tcp arrived at 1784197964 with bad export cookie 6461899022330875894 [13076.326586] LustreError: 423250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13078.018448] LustreError: 423251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197965 with bad export cookie 6461899022330875229 [13078.033053] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13078.496986] LustreError: 423270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13078.505014] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13078.521478] LustreError: 423270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13078.547419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13083.620366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13083.625098] LustreError: 424327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13083.663414] LustreError: 424327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13084.291940] Lustre: server umount lustre-OST0000 complete [13087.717331] LustreError: 423270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13087.742119] LustreError: 423270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13090.516983] Lustre: server umount lustre-MDT0001 complete [13100.111496] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13102.214491] Key type lgssc unregistered [13102.427823] LNet: 426956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13102.439158] LNetError: 426956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13102.452363] LNet: Removed LNI 192.168.204.120@tcp [13103.180761] Key type .llcrypt unregistered [13103.183409] Key type ._llcrypt unregistered [13122.224723] Key type ._llcrypt registered [13122.228657] Key type .llcrypt registered [13122.315366] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13123.068901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13123.125644] alg: No test for adler32 (adler32-zlib) [13124.131734] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [13124.341784] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13125.983190] Key type lgssc registered [13126.865581] Lustre: Echo OBD driver; http://www.lustre.org/ [13133.051140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13133.078655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13134.415843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13138.008715] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13144.550864] LustreError: 428351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13144.564338] LustreError: 428351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13145.180811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13145.496284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13148.793348] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13154.928555] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13159.091544] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13164.205688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13164.438693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13168.655378] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13174.687433] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13179.825575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13184.998737] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13185.000960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13185.008804] Lustre: Skipped 1 previous similar message [13190.118989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13190.133701] Lustre: Skipped 3 previous similar messages [13195.250840] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13195.259296] Lustre: Skipped 2 previous similar messages [13196.255111] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13196.427538] Lustre: server umount lustre-MDT0001 complete [13200.352745] LustreError: 429079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13200.369392] LustreError: 429079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13205.480873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13205.481304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13205.481983] LustreError: 428352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13205.491764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13205.515413] Lustre: Skipped 1 previous similar message [13206.675622] Lustre: server umount lustre-OST0000 complete [13209.750195] Lustre: server umount lustre-MDT0000 complete [13221.414347] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13223.842264] Key type lgssc unregistered [13224.106872] LNet: 432036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13224.113869] LNetError: 432036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13224.136494] LNet: Removed LNI 192.168.204.120@tcp [13224.795095] Key type .llcrypt unregistered [13224.797235] Key type ._llcrypt unregistered [13246.819621] Key type ._llcrypt registered [13246.821749] Key type .llcrypt registered [13246.934751] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [13259.455818] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13260.377647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13260.458817] alg: No test for adler32 (adler32-zlib) [13261.547547] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [13261.719094] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13263.351213] Key type lgssc registered [13264.292801] Lustre: Echo OBD driver; http://www.lustre.org/ [13296.640845] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13309.727627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13309.757174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13310.975768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13310.997699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13311.055905] Lustre: lustre-MDT0000: new disk, initializing [13311.108432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13311.124431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13313.904952] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13323.865450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13324.002808] Lustre: lustre-OST0000: new disk, initializing [13324.005419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13324.007329] Lustre: Skipped 1 previous similar message [13324.010163] Lustre: 437049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13324.055860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13327.542652] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13331.504687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13331.509766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13331.537536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13333.526898] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13341.241849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13341.309094] Lustre: 436031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13341.343208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13341.382693] Lustre: lustre-MDT0001: new disk, initializing [13341.438543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13341.455781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13341.458609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13343.553162] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13347.979145] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13348.067576] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13349.714677] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13349.802085] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13352.385363] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [13354.469455] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13354.473913] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13354.475168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13354.481638] Lustre: Skipped 1 previous similar message [13357.479620] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [13357.488216] Lustre: Skipped 1 previous similar message [13358.362787] Lustre: server umount lustre-MDT0001 complete [13364.715752] LustreError: 436038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13364.734075] LustreError: 436038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13369.831716] LustreError: 438373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13369.859644] LustreError: 438373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13374.946185] LustreError: 436036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13374.963257] LustreError: 436036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13377.505263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13377.510163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13377.517567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13380.072264] LustreError: 438373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13381.273821] Lustre: server umount lustre-OST0000 complete [13390.713700] Lustre: server umount lustre-MDT0000 complete [13401.752461] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13403.959200] Key type lgssc unregistered [13404.201756] LNet: 440792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13404.220829] LNetError: 440792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13404.242228] LNet: Removed LNI 192.168.204.120@tcp [13404.907380] Key type .llcrypt unregistered [13404.910596] Key type ._llcrypt unregistered [13435.104586] Key type ._llcrypt registered [13435.107796] Key type .llcrypt registered [13435.198495] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13435.412912] Key type .llcrypt unregistered [13435.415592] Key type ._llcrypt unregistered [13447.570811] Key type ._llcrypt registered [13447.575455] Key type .llcrypt registered [13447.719915] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13448.476858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13448.493180] alg: No test for adler32 (adler32-zlib) [13449.458695] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [13449.616597] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13451.271179] Key type lgssc registered [13452.142513] Lustre: Echo OBD driver; http://www.lustre.org/ [13476.855366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13476.902553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13478.184249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13478.249053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13478.368188] Lustre: lustre-MDT0000: new disk, initializing [13478.455551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13478.475617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13483.033931] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13492.910219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13493.015596] Lustre: 444148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13493.077864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13493.087267] Lustre: Skipped 1 previous similar message [13493.157110] Lustre: lustre-MDT0001: new disk, initializing [13493.246516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13493.278059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13493.295763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13496.949870] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13504.592839] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13509.165251] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13515.094793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13515.292971] Lustre: lustre-OST0000: new disk, initializing [13515.296993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13515.305887] Lustre: 446116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13515.397665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13520.642724] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13525.034617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13525.049552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13525.143922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13528.342335] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13534.176179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13534.182888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13534.199614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13535.713481] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13535.715314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13535.734577] Lustre: Skipped 1 previous similar message [13535.736915] Lustre: Skipped 1 previous similar message [13539.839123] Lustre: server umount lustre-MDT0000 complete [13543.020485] LustreError: 444138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198430 with bad export cookie 4598423134846763019 [13543.031793] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13543.033484] LustreError: 444138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13543.287701] Lustre: server umount lustre-MDT0001 complete [13557.170401] Lustre: server umount lustre-OST0000 complete [13567.111935] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:40:52 (1784198452) [13573.619733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13573.941136] LustreError: 448140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13573.961777] LustreError: 448140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13574.011589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13577.503485] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13579.233854] LustreError: 448141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13583.330077] LustreError: 448140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13584.401151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13584.786493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13588.740703] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13595.479597] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13600.208985] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13618.804047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13618.969162] Lustre: Found index 0 for lustre-OST0000, updating log [13619.108857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13624.057217] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13629.934978] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13644.719676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13649.889269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13649.890352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13649.892419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13649.892427] Lustre: Skipped 1 previous similar message [13649.903292] Lustre: Skipped 1 previous similar message [13651.202731] Lustre: server umount lustre-OST0000 complete [13666.795062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13666.807603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13666.826329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13666.832156] Lustre: Skipped 3 previous similar messages [13672.154429] Lustre: server umount lustre-MDT0000 complete [13672.418950] LustreError: 448867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13672.428462] LustreError: 448867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13675.327139] LustreError: 448122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198563 with bad export cookie 4598423134846764412 [13675.329284] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13675.343661] LustreError: 448122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13675.609827] Lustre: server umount lustre-MDT0001 complete [13682.709554] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:42:48 (1784198568) [13687.199507] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [14280.855832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14281.115910] LustreError: 454566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14281.157979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14283.153874] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14287.507480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14287.690158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14289.621238] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14293.047428] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14295.532642] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14311.845260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14311.948464] Lustre: Found index 0 for lustre-OST0000, updating log [14312.063622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14314.576959] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14317.767344] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14320.609706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14320.612943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14320.617244] Lustre: Skipped 1 previous similar message [14320.625243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14320.628352] Lustre: Skipped 2 previous similar messages [14324.767627] Lustre: server umount lustre-OST0000 complete [14326.340415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14326.341492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14326.345803] Lustre: Skipped 1 previous similar message [14327.392478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14327.395682] Lustre: Skipped 2 previous similar messages [14329.440282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14329.443909] Lustre: Skipped 3 previous similar messages [14332.512279] LustreError: 455295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14332.519194] LustreError: 455295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14332.528285] Lustre: server umount lustre-MDT0000 complete [14333.968322] LustreError: 454546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784199221 with bad export cookie 4598423134846765497 [14333.971500] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14333.972732] LustreError: 454546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14334.098179] Lustre: server umount lustre-MDT0001 complete [14340.511263] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [14352.904908] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [14357.157143] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [14371.713500] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:54:17 (1784199257) [14378.575306] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [14379.791568] Key type lgssc unregistered [14379.959959] LNet: 461200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14379.967288] LNetError: 461200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14444.517353] LNet: Removed LNI 192.168.204.120@tcp [14444.871961] Key type .llcrypt unregistered [14444.873139] Key type ._llcrypt unregistered [14452.244147] Key type ._llcrypt registered [14452.245528] Key type .llcrypt registered [14452.291909] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [14452.630693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14452.669053] alg: No test for adler32 (adler32-zlib) [14453.559460] Lustre: Lustre: Build Version: 2.17.54_162_gdc75e25 [14453.670536] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [14455.271234] Key type lgssc registered [14455.750775] Lustre: Echo OBD driver; http://www.lustre.org/ [14457.690677] LNet: Added LNI 192.168.204.120@tcp100 [8/256/0/180] [14458.089913] LNet: Added LNI 192.168.204.120@tcp101 [8/256/0/180] [14458.493756] LNet: Added LNI 192.168.204.120@tcp102 [8/256/0/180] [14458.941814] LNet: Added LNI 192.168.204.120@tcp103 [8/256/0/180] [14459.314815] LNet: Added LNI 192.168.204.120@tcp104 [8/256/0/180] [14459.675748] LNet: Added LNI 192.168.204.120@tcp105 [8/256/0/180] [14460.037774] LNet: Added LNI 192.168.204.120@tcp106 [8/256/0/180] [14460.401155] LNet: Added LNI 192.168.204.120@tcp107 [8/256/0/180] [14460.772140] LNet: Added LNI 192.168.204.120@tcp108 [8/256/0/180] [14461.122666] LNet: Added LNI 192.168.204.120@tcp109 [8/256/0/180] [14461.482336] LNet: Added LNI 192.168.204.120@tcp110 [8/256/0/180] [14461.827570] LNet: Added LNI 192.168.204.120@tcp111 [8/256/0/180] [14462.171033] LNet: Added LNI 192.168.204.120@tcp112 [8/256/0/180] [14462.510874] LNet: Added LNI 192.168.204.120@tcp113 [8/256/0/180] [14462.848281] LNet: Added LNI 192.168.204.120@tcp114 [8/256/0/180] [14463.179484] LNet: Added LNI 192.168.204.120@tcp115 [8/256/0/180] [14463.503479] LNet: Added LNI 192.168.204.120@tcp116 [8/256/0/180] [14463.857368] LNet: Added LNI 192.168.204.120@tcp117 [8/256/0/180] [14464.198251] LNet: Added LNI 192.168.204.120@tcp118 [8/256/0/180] [14464.545297] LNet: Added LNI 192.168.204.120@tcp119 [8/256/0/180] [14464.882896] LNet: Added LNI 192.168.204.120@tcp120 [8/256/0/180] [14465.239541] LNet: Added LNI 192.168.204.120@tcp121 [8/256/0/180] [14465.587837] LNet: Added LNI 192.168.204.120@tcp122 [8/256/0/180] [14465.932692] LNet: Added LNI 192.168.204.120@tcp123 [8/256/0/180] [14466.283864] LNet: Added LNI 192.168.204.120@tcp124 [8/256/0/180] [14466.662604] LNet: Added LNI 192.168.204.120@tcp125 [8/256/0/180] [14467.014785] LNet: Added LNI 192.168.204.120@tcp126 [8/256/0/180] [14467.387061] LNet: Added LNI 192.168.204.120@tcp127 [8/256/0/180] [14467.767237] LNet: Added LNI 192.168.204.120@tcp128 [8/256/0/180] [14468.129418] LNet: Added LNI 192.168.204.120@tcp129 [8/256/0/180] [14468.498607] LNet: Added LNI 192.168.204.120@tcp130 [8/256/0/180] [14468.840765] LNet: Added LNI 192.168.204.120@tcp131 [8/256/0/180] [14469.189683] LNet: Added LNI 192.168.204.120@tcp132 [8/256/0/180] [14469.544670] LNet: Added LNI 192.168.204.120@tcp133 [8/256/0/180] [14469.911897] LNet: Added LNI 192.168.204.120@tcp134 [8/256/0/180] [14470.275256] LNet: Added LNI 192.168.204.120@tcp135 [8/256/0/180] [14473.084060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14473.090522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14474.137026] Lustre: 464203:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [14474.160854] LNetError: 464225:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [14474.166152] LNetError: Protocol error connecting to 192.168.204.120@tcp116 at host 192.168.204.120:988: is it running a compatible version of Lustre? [14474.213219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14474.227946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14474.269253] Lustre: lustre-MDT0000: new disk, initializing [14474.307606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14474.315508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14475.168964] LNetError: 461666:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [14475.174558] LNetError: Protocol error connecting to 192.168.204.120@tcp114 at host 192.168.204.120:988: is it running a compatible version of Lustre? [14475.876408] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14480.698639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14480.760796] Lustre: 464220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14480.795353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14480.797936] Lustre: Skipped 1 previous similar message [14480.839356] Lustre: lustre-MDT0001: new disk, initializing [14480.854691] Lustre: 465023:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [14480.859355] Lustre: 465023:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14480.907393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14480.922892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14480.927140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14482.607250] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14486.008824] Lustre: 465553:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [14486.015711] Lustre: 465553:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14487.101927] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14489.438771] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14492.270406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14492.400875] Lustre: lustre-OST0000: new disk, initializing [14492.403800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14492.407996] Lustre: 466196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14492.449466] Lustre: 466201:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [14492.453581] Lustre: 466201:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14492.475054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14494.803178] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14499.212809] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14502.278152] Lustre: server umount lustre-OST0000 complete [14502.395640] Lustre: 467155:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [14502.400525] Lustre: 467155:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14506.464181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14506.467930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14506.473876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14507.488398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14507.488682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14509.985775] LNetError: 464227:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [14509.992110] LNetError: Protocol error connecting to 192.168.204.120@tcp116 at host 192.168.204.120:988: is it running a compatible version of Lustre? [14510.015694] Lustre: server umount lustre-MDT0000 complete [14511.584620] LustreError: 465022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784199399 with bad export cookie 8732600237242261506 [14511.587897] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14511.597179] LustreError: 465022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14511.729112] Lustre: server umount lustre-MDT0001 complete [14512.032642] LNetError: 461668:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [14512.037966] LNetError: Protocol error connecting to 192.168.204.120@tcp114 at host 192.168.204.120:988: is it running a compatible version of Lustre? [14515.670629] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [14516.876953] Key type lgssc unregistered [14517.063445] LNet: 468105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14517.067814] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14517.071778] LNet: Removed LNI 192.168.204.120@tcp [14518.111154] LNet: Removed LNI 192.168.204.120@tcp100 [14518.112976] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14518.115790] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14518.118801] LNet: Removed LNI 192.168.204.120@tcp101 [14519.135147] LNet: Removed LNI 192.168.204.120@tcp102 [14519.136832] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14519.140398] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14519.143583] LNet: Removed LNI 192.168.204.120@tcp103 [14520.159462] LNet: Removed LNI 192.168.204.120@tcp104 [14521.184649] LNet: Removed LNI 192.168.204.120@tcp105 [14521.187193] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14521.190070] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14521.194039] LNet: Removed LNI 192.168.204.120@tcp106 [14522.208511] LNet: Removed LNI 192.168.204.120@tcp107 [14523.231142] LNet: Removed LNI 192.168.204.120@tcp108 [14524.255159] LNet: Removed LNI 192.168.204.120@tcp109 [14525.279146] LNet: Removed LNI 192.168.204.120@tcp110 [14525.281451] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14525.285723] LNetError: 468105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14525.290000] LNet: Removed LNI 192.168.204.120@tcp111 [14526.303120] LNet: Removed LNI 192.168.204.120@tcp112 [14527.327180] LNet: Removed LNI 192.168.204.120@tcp113 [14528.351134] LNet: Removed LNI 192.168.204.120@tcp114 [14529.375155] LNet: Removed LNI 192.168.204.120@tcp115 [14530.399399] LNet: Removed LNI 192.168.204.120@tcp116 [14531.423294] LNet: Removed LNI 192.168.204.120@tcp117 [14531.425198] LNet: Removed LNI 192.168.204.120@tcp118 [14531.427260] LNet: Removed LNI 192.168.204.120@tcp119 [14531.432100] LNet: Removed LNI 192.168.204.120@tcp120 [14531.436394] LNet: Removed LNI 192.168.204.120@tcp121 [14531.441100] LNet: Removed LNI 192.168.204.120@tcp122 [14531.444761] LNet: Removed LNI 192.168.204.120@tcp123 [14531.446788] LNet: Removed LNI 192.168.204.120@tcp124 [14531.449360] LNet: Removed LNI 192.168.204.120@tcp125 [14531.452088] LNet: Removed LNI 192.168.204.120@tcp126 [14531.455549] LNet: Removed LNI 192.168.204.120@tcp127 [14531.458150] LNet: Removed LNI 192.168.204.120@tcp128 [14531.460539] LNet: Removed LNI 192.168.204.120@tcp129 [14531.462895] LNet: Removed LNI 192.168.204.120@tcp130 [14531.465265] LNet: Removed LNI 192.168.204.120@tcp131 [14531.467722] LNet: Removed LNI 192.168.204.120@tcp132 [14531.470623] LNet: Removed LNI 192.168.204.120@tcp133 [14531.473645] LNet: Removed LNI 192.168.204.120@tcp134 [14531.482970] LNet: Removed LNI 192.168.204.120@tcp135 [14531.854075] Key type .llcrypt unregistered [14531.855790] Key type ._llcrypt unregistered