[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 435571818 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003232] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008350] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011171] LSM: Security Framework initializing [ 0.012039] Yama: becoming mindful. [ 0.013059] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.022540] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027435] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029089] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030141] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032022] *** VALIDATE tmpfs *** [ 0.033525] *** VALIDATE proc *** [ 0.035187] *** VALIDATE cgroup *** [ 0.036024] *** VALIDATE cgroup2 *** [ 0.037273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038200] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040039] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.045475] debug: unmapping init [mem 0xffffffffa2859000-0xffffffffa2860fff] [ 0.048294] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049825] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050018] ... version: 2 [ 0.051009] ... bit width: 48 [ 0.052006] ... generic registers: 4 [ 0.053006] ... value mask: 0000ffffffffffff [ 0.054010] ... max period: 00007fffffffffff [ 0.055010] ... fixed-purpose events: 3 [ 0.056006] ... event mask: 000000070000000f [ 0.057313] rcu: Hierarchical SRCU implementation. [ 0.059478] smp: Bringing up secondary CPUs ... [ 0.060493] x86: Booting SMP configuration: [ 0.061018] .... node #0, CPUs: #1 #2 #3 [ 0.067020] smp: Brought up 1 node, 4 CPUs [ 0.069010] smpboot: Max logical packages: 1 [ 0.070013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.203361] node 0 deferred pages initialised in 131ms [ 0.207224] devtmpfs: initialized [ 0.208269] x86/mm: Memory block size: 128MB [ 0.211082] gcov: version magic: 0x41383552 [ 0.213205] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.214074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.215267] pinctrl core: initialized pinctrl subsystem [ 0.216191] [ 0.216586] ************************************************************* [ 0.217029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.218014] ** ** [ 0.219014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.220015] ** ** [ 0.221031] ** This means that this kernel is built to expose internal ** [ 0.222013] ** IOMMU data structures, which may compromise security on ** [ 0.223014] ** your system. ** [ 0.224009] ** ** [ 0.225013] ** If you see this message and you are not debugging the ** [ 0.226014] ** kernel, report this immediately to your vendor! ** [ 0.227016] ** ** [ 0.228009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229012] ************************************************************* [ 0.230848] NET: Registered protocol family 16 [ 0.231418] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.232061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.233051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.235079] cpuidle: using governor menu [ 0.239100] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.241450] PCI: Using configuration type 1 for base access [ 0.243089] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.250110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.251018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.253054] cryptd: max_cpu_qlen set to 1000 [ 0.256069] ACPI: Added _OSI(Module Device) [ 0.257011] ACPI: Added _OSI(Processor Device) [ 0.258009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.259014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.263195] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.266626] ACPI: Interpreter enabled [ 0.267058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.268011] ACPI: Using IOAPIC for interrupt routing [ 0.269130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.279427] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.280054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.281022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284493] acpiphp: Slot [2] registered [ 0.285127] acpiphp: Slot [5] registered [ 0.286109] acpiphp: Slot [6] registered [ 0.287085] acpiphp: Slot [7] registered [ 0.288074] acpiphp: Slot [8] registered [ 0.289083] acpiphp: Slot [9] registered [ 0.290065] acpiphp: Slot [10] registered [ 0.291000] acpiphp: Slot [3] registered [ 0.291000] acpiphp: Slot [4] registered [ 0.293143] acpiphp: Slot [11] registered [ 0.294088] acpiphp: Slot [12] registered [ 0.296080] acpiphp: Slot [13] registered [ 0.297076] acpiphp: Slot [14] registered [ 0.298077] acpiphp: Slot [15] registered [ 0.299076] acpiphp: Slot [16] registered [ 0.301073] acpiphp: Slot [17] registered [ 0.302096] acpiphp: Slot [18] registered [ 0.303020] acpiphp: Slot [19] registered [ 0.304068] acpiphp: Slot [20] registered [ 0.306089] acpiphp: Slot [21] registered [ 0.307032] acpiphp: Slot [22] registered [ 0.308056] acpiphp: Slot [23] registered [ 0.309086] acpiphp: Slot [24] registered [ 0.310078] acpiphp: Slot [25] registered [ 0.311058] acpiphp: Slot [26] registered [ 0.312082] acpiphp: Slot [27] registered [ 0.313099] acpiphp: Slot [28] registered [ 0.315088] acpiphp: Slot [29] registered [ 0.316198] acpiphp: Slot [30] registered [ 0.318088] acpiphp: Slot [31] registered [ 0.319092] PCI host bridge to bus 0000:00 [ 0.320019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.328024] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.330014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333017] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.336064] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344832] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.347654] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357301] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359718] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.364765] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.370016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.383020] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.388013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391822] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.405018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.418025] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.434076] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.441037] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.461022] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.472061] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.478017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.486022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.499026] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.511000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516048] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.525026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.539057] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.551908] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.557021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.565025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.579021] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.587317] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.594019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.598016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.612021] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.623918] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.626361] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.628323] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.630324] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.632195] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.635120] iommu: Default domain type: Passthrough [ 0.637413] SCSI subsystem initialized [ 0.638102] ACPI: bus type USB registered [ 0.640091] usbcore: registered new interface driver usbfs [ 0.641067] usbcore: registered new interface driver hub [ 0.643065] usbcore: registered new device driver usb [ 0.644179] pps_core: LinuxPPS API ver. 1 registered [ 0.646014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.649045] PTP clock support registered [ 0.651220] EDAC MC: Ver: 3.0.0 [ 0.653153] PCI: Using ACPI for IRQ routing [ 0.654794] NetLabel: Initializing [ 0.655011] NetLabel: domain hash size = 128 [ 0.656022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.657178] NetLabel: unlabeled traffic allowed by default [ 0.660096] vgaarb: loaded [ 0.661298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.663018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.670482] clocksource: Switched to clocksource kvm-clock [ 0.770648] VFS: Disk quotas dquot_6.6.0 [ 0.772383] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.775152] *** VALIDATE ramfs *** [ 0.776165] *** VALIDATE hugetlbfs *** [ 0.777339] pnp: PnP ACPI init [ 0.779089] pnp: PnP ACPI: found 6 devices [ 0.796396] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.798923] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.800383] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.801614] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.802865] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.804375] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.806019] NET: Registered protocol family 2 [ 0.808084] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.811683] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.814873] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.820837] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.825570] TCP: Hash tables configured (established 65536 bind 65536) [ 0.829276] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.831870] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.834379] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.837323] NET: Registered protocol family 1 [ 0.839628] RPC: Registered named UNIX socket transport module. [ 0.841935] RPC: Registered udp transport module. [ 0.843499] RPC: Registered tcp transport module. [ 0.845311] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.847719] NET: Registered protocol family 44 [ 0.849458] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.851884] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.854152] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.855644] PCI: CLS 0 bytes, default 64 [ 0.856876] Unpacking initramfs... [ 2.462372] debug: unmapping init [mem 0xffff8eeefcc54000-0xffff8eeefffbffff] [ 2.466987] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.469025] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.471622] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.194894] Initialise system trusted keyrings [ 3.200370] Key type blacklist registered [ 3.205100] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.230625] zbud: loaded [ 3.242156] *** VALIDATE nfs *** [ 3.243835] *** VALIDATE nfs4 *** [ 3.246131] pstore: using deflate compression [ 3.251904] Platform Keyring initialized [ 3.472818] NET: Registered protocol family 38 [ 3.475987] Key type asymmetric registered [ 3.478412] Asymmetric key parser 'x509' registered [ 3.481285] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.493187] io scheduler mq-deadline registered [ 3.495475] io scheduler kyber registered [ 3.497138] io scheduler bfq registered [ 3.498718] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.502068] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.504696] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.508088] ACPI: Power Button [PWRF] [ 3.611404] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.706819] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.913384] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.033270] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.245426] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.278739] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.312323] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.316813] Non-volatile memory driver v1.3 [ 4.318179] Linux agpgart interface v0.103 [ 4.346886] virtio_blk virtio1: [vda] 133640 512-byte logical blocks (68.4 MB/65.3 MiB) [ 4.349934] vda: detected capacity change from 0 to 68423680 [ 4.373387] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.376419] vdb: detected capacity change from 0 to 1073741824 [ 4.406154] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.409629] vdc: detected capacity change from 0 to 2621440000 [ 4.432460] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.435395] vdd: detected capacity change from 0 to 2621440000 [ 4.458358] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.462058] vde: detected capacity change from 0 to 4294967296 [ 4.481158] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.484380] vdf: detected capacity change from 0 to 4294967296 [ 4.490685] libphy: Fixed MDIO Bus: probed [ 4.496427] usbcore: registered new interface driver usbserial_generic [ 4.498747] usbserial: USB Serial support registered for generic [ 4.501357] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.505707] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.508371] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.510892] mousedev: PS/2 mouse device common for all mice [ 4.514489] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.518363] rtc_cmos 00:05: RTC can wake from S4 [ 4.525047] rtc_cmos 00:05: registered as rtc0 [ 4.527259] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.529821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.536361] intel_pstate: CPU model not supported [ 4.542501] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.547552] hid: raw HID events driver (C) Jiri Kosina [ 4.549422] usbcore: registered new interface driver usbhid [ 4.551552] usbhid: USB HID core driver [ 4.553159] drop_monitor: Initializing network drop monitor service [ 4.555753] Initializing XFRM netlink socket [ 4.557810] NET: Registered protocol family 10 [ 4.561834] Segment Routing with IPv6 [ 4.563253] NET: Registered protocol family 17 [ 4.570726] mpls_gso: MPLS GSO support [ 4.576912] RAS: Correctable Errors collector initialized. [ 4.578809] AVX version of gcm_enc/dec engaged. [ 4.580271] AES CTR mode by8 optimization enabled [ 4.688502] sched_clock: Marking stable (4688016258, 0)->(5534484887, -846468629) [ 4.691573] registered taskstats version 1 [ 4.694284] Loading compiled-in X.509 certificates [ 4.696245] zswap: loaded using pool lzo/zbud [ 4.729856] Key type big_key registered [ 4.745364] Key type encrypted registered [ 4.746747] ima: No TPM chip found, activating TPM-bypass! [ 4.749161] ima: Allocated hash algorithm: sha1 [ 4.752198] ima: No architecture policies found [ 4.753771] evm: Initialising EVM extended attributes: [ 4.755641] evm: security.selinux [ 4.756810] evm: security.ima [ 4.757876] evm: security.capability [ 4.759324] evm: HMAC attrs: 0x1 [ 4.762437] rtc_cmos 00:05: setting system clock to 2025-12-03 19:41:50 UTC (1764790910) [ 4.769258] debug: unmapping init [mem 0xffffffffa3803000-0xffffffffa39fffff] [ 4.772656] debug: unmapping init [mem 0xffffffffa2582000-0xffffffffa2858fff] [ 4.781045] Write protecting the kernel read-only data: 28672k [ 4.784361] debug: unmapping init [mem 0xffffffffa0c03000-0xffffffffa0dfffff] [ 4.786900] debug: unmapping init [mem 0xffffffffa1514000-0xffffffffa15fffff] [ 4.822442] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.830688] systemd[1]: Detected virtualization kvm. [ 4.832335] systemd[1]: Detected architecture x86-64. [ 4.834762] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.864324] systemd[1]: No hostname configured. [ 4.865803] systemd[1]: Set hostname to . [ 4.867754] random: systemd: uninitialized urandom read (16 bytes read) [ 4.870291] systemd[1]: Initializing machine ID from random generator. [ 5.119674] random: systemd: uninitialized urandom read (16 bytes read) [ 5.122498] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.128192] random: systemd: uninitialized urandom read (16 bytes read) [ 5.130830] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.136773] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.326692] device-mapper: uevent: version 1.0.3 [ 6.333690] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.360240] random: fast init done Starting dracut initqueue hook... [ 7.418462] virtio_net virtio0 ens2: renamed from eth0 [ 7.719958] scsi host0: ata_piix [ 7.820117] scsi host1: ata_piix [ 7.831235] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.833639] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.282056] random: crng init done [ 12.283402] random: 7 urandom warning(s) missed due to ratelimiting [ 14.002156] dracut-initqueue[586]: RTNETLINK answers: File exists [ 14.299780] hrtimer: interrupt took 4940638 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.279606] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.973635] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.374127] SELinux: Disabled at runtime. [ 18.443061] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.452917] systemd[1]: Detected virtualization kvm. [ 18.454798] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.395168] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.398293] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.405414] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.410681] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.414837] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.436339] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.451596] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ 19.652291] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.363511] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.055497] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.127474] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.667326] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.397784] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 27.067497] Key type dns_resolver registered [ 27.489646] NFS: Registering the id_resolver key type [ 27.495824] Key type id_resolver registered [ 27.498122] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-server login: [ 55.139523] libcfs: loading out-of-tree module taints kernel. [ 55.150040] Key type ._llcrypt registered [ 55.150902] Key type .llcrypt registered [ 55.187564] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 62.263336] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 62.811597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.817806] alg: No test for adler32 (adler32-zlib) [ 63.818360] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 64.113181] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 65.728158] Key type lgssc registered [ 66.304630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.531112] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.246391] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.494025] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.257756] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.978895] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.158279] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 93.124924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.145781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 93.151858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.238670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 94.253360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 94.299054] Lustre: lustre-MDT0000: new disk, initializing [ 94.332173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.340287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 95.724417] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 100.853886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.877165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 100.904275] Lustre: 6483:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 100.917975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 100.919775] Lustre: Skipped 1 previous similar message [ 100.956342] Lustre: lustre-MDT0001: new disk, initializing [ 100.980971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.992158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 100.995933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 102.270199] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.579890] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 107.715319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.739747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 107.826893] Lustre: lustre-OST0000: new disk, initializing [ 107.828576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 107.844199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.673947] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.677486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 109.680103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 109.707891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 114.927741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 114.958315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 114.997215] Lustre: lustre-OST0001: new disk, initializing [ 114.998755] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 115.001340] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 115.021950] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.940538] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.292530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 120.295286] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 120.307294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 123.076373] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 128.590875] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 137.652248] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing check_logdir /tmp/testlogs/ [ 140.792278] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing yml_node [ 143.685842] Lustre: DEBUG MARKER: Client: 2.16.59.23 [ 145.384839] Lustre: DEBUG MARKER: MDS: 2.16.59.23 [ 147.260840] Lustre: DEBUG MARKER: OSS: 2.16.59.23 [ 148.450783] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Dec 3 14:44:12 EST 2025 [ 161.677911] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 162.567638] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 176.608871] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 176.609193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 176.616794] Lustre: Skipped 2 previous similar messages [ 176.619103] Lustre: Skipped 1 previous similar message [ 181.056512] LustreError: 12193:0:(obd_class.h:479:obd_check_dev()) Device 26 not setup [ 181.081674] Lustre: server umount lustre-MDT0000 complete [ 182.472704] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791088 with bad export cookie 5486636384149645885 [ 182.474665] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 182.476160] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 186.848875] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 186.849621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 186.850033] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 186.850036] Lustre: Skipped 2 previous similar messages [ 186.854783] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 186.858550] Lustre: Skipped 2 previous similar messages [ 187.872630] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 187.878728] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 188.611490] LustreError: 12396:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 188.613866] LustreError: 12396:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 188.691328] Lustre: server umount lustre-MDT0001 complete [ 190.151229] LustreError: 12597:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 190.155640] LustreError: 12597:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 190.182746] Lustre: server umount lustre-OST0000 complete [ 191.554685] Lustre: server umount lustre-OST0001 complete [ 193.744810] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 197.424061] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 201.301177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 204.041972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.534469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.261497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.085817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.110987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.217777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 212.238272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 212.281689] Lustre: lustre-MDT0000: new disk, initializing [ 212.325408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.334887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.941245] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 219.013045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 219.045199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 219.094027] Lustre: 14975:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.229654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.859143] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 225.412715] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 228.805645] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 235.000392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 235.063379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 235.274685] Lustre: lustre-OST0000: new disk, initializing [ 235.282623] Lustre: Skipped 1 previous similar message [ 235.294716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.302528] Lustre: Skipped 2 previous similar messages [ 235.422304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.427775] Lustre: Skipped 1 previous similar message [ 236.639840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.648625] Lustre: Skipped 1 previous similar message [ 236.651856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.749797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.580800] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 248.031603] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 251.500534] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 251.742539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 255.503746] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 255.754988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 257.506931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 257.519353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.526701] LustreError: Skipped 1 previous similar message [ 257.527906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 257.552149] Lustre: Skipped 1 previous similar message [ 257.574199] Lustre: Skipped 4 previous similar messages [ 262.626103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 262.633833] Lustre: Skipped 1 previous similar message [ 263.497604] LustreError: 17917:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 263.507032] LustreError: 17917:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 263.659349] Lustre: server umount lustre-OST0000 complete [ 267.745923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.772715] Lustre: Skipped 1 previous similar message [ 272.838700] LustreError: 18118:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 272.842858] LustreError: 18118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 272.867041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 272.884217] Lustre: Skipped 2 previous similar messages [ 272.895627] LustreError: 14981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.993259] Lustre: server umount lustre-MDT0000 complete [ 276.668129] LustreError: 14966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791182 with bad export cookie 5486636384149647348 [ 276.670845] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 276.685945] LustreError: 14966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 277.090952] Lustre: server umount lustre-MDT0001 complete [ 280.244949] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:46:24 (1764791184) [ 281.644251] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 282.783173] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:46:27 (1764791187) [ 287.283587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 287.614363] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.681287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 292.535503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 292.839753] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.084832] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.096961] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 303.391232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 309.901701] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 317.548837] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 323.196968] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 330.263947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 330.576401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 330.579040] Lustre: Skipped 1 previous similar message [ 335.340952] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 339.677478] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 354.276098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 354.277200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.279047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 354.286407] Lustre: Skipped 1 previous similar message [ 355.721460] LustreError: 21260:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 355.728750] LustreError: 21260:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 355.887696] Lustre: server umount lustre-OST0000 complete [ 373.216172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 373.277251] LustreError: 21462:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 373.282946] LustreError: 21462:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 373.333959] Lustre: server umount lustre-MDT0000 complete [ 374.765279] LustreError: 18988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.782023] LustreError: 18988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 375.910132] LustreError: 18969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791281 with bad export cookie 5486636384149648230 [ 375.920549] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.926052] LustreError: 18969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 376.190821] Lustre: server umount lustre-MDT0001 complete [ 382.716371] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 384.453266] Key type lgssc unregistered [ 384.641903] LNet: 22197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.645643] LNetError: 22197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 384.656936] LNet: Removed LNI 192.168.201.105@tcp [ 385.193240] Key type .llcrypt unregistered [ 385.194663] Key type ._llcrypt unregistered [ 397.931423] Key type ._llcrypt registered [ 397.933721] Key type .llcrypt registered [ 397.997188] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 398.485289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 398.498563] alg: No test for adler32 (adler32-zlib) [ 399.437494] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 399.587910] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 401.208244] Key type lgssc registered [ 401.930782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 407.241688] loop: module loaded [ 407.274406] loop0: detected capacity change from 0 to 204800000 [ 407.371943] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 407.378375] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 407.572775] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 407.612734] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 411.098652] loop0: detected capacity change from 0 to 204800000 [ 411.213581] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 411.225437] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 411.235459] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 411.329546] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 414.083835] loop0: detected capacity change from 0 to 204800000 [ 419.366359] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 419.452842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 419.477465] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.737853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 420.775749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 420.862307] Lustre: lustre-MDT0000: new disk, initializing [ 421.014156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.073096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.094364] Lustre: 24309:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 466, rollback = 2 [ 421.099366] Lustre: 24309:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 421.102495] Lustre: 24309:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/466/0 [ 421.105971] Lustre: 24309:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 421.110030] Lustre: 24309:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 421.114161] Lustre: 24309:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 426.406982] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 438.140776] loop1: detected capacity change from 0 to 204800000 [ 448.072719] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 448.197439] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 448.996606] Lustre: lustre-OST07c6: new disk, initializing [ 449.005041] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 449.009914] Lustre: Skipped 1 previous similar message [ 449.149531] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 455.861920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 455.882916] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 455.952386] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 457.677464] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 464.762272] Lustre: 24333:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 464.778120] Lustre: 24333:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 464.787173] Lustre: 24333:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 464.805870] Lustre: 24333:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 464.825355] Lustre: 24333:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 464.839846] Lustre: 24333:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 464.858812] Lustre: 24333:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 464.871310] Lustre: 24333:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 464.884786] Lustre: 24333:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 464.898939] Lustre: 24333:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 464.907622] Lustre: 24333:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 464.919799] Lustre: 24333:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 482.150945] Lustre: 24335:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 482.167760] Lustre: 24335:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 482.183024] Lustre: 24335:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 482.188159] Lustre: 24335:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 482.197220] Lustre: 24335:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 482.204517] Lustre: 24335:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 487.181069] Lustre: 24335:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 487.185327] Lustre: 24335:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 487.188768] Lustre: 24335:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 487.191704] Lustre: 24335:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.194925] Lustre: 24335:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 487.198518] Lustre: 24335:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.202037] Lustre: 24335:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 487.211877] Lustre: 24335:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.218853] Lustre: 24335:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 487.224088] Lustre: 24335:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 487.229443] Lustre: 24335:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 487.233794] Lustre: 24335:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 491.799973] Lustre: 24335:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 497, rollback = 2 [ 491.818339] Lustre: 24335:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 491.825730] Lustre: 24335:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 491.843960] Lustre: 24335:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 491.850622] Lustre: 24335:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 491.859333] Lustre: 24335:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 491.864265] Lustre: 24335:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 491.869555] Lustre: 24335:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 491.883023] Lustre: 24335:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 491.888438] Lustre: 24335:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 491.893144] Lustre: 24335:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 491.897525] Lustre: 24335:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 502.753482] Lustre: 24334:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 502.762702] Lustre: 24334:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 5 previous similar messages [ 502.768363] Lustre: 24334:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 502.773864] Lustre: 24334:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 502.789556] Lustre: 24334:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 502.796144] Lustre: 24334:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 502.810562] Lustre: 24334:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 502.814722] Lustre: 24334:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 502.824915] Lustre: 24334:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 502.828201] Lustre: 24334:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 502.831666] Lustre: 24334:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 502.834914] Lustre: 24334:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 558.049719] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.065491] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 560.368431] LustreError: 26522:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 560.508615] Lustre: server umount lustre-OST07c6 complete [ 564.989867] LustreError: 27023:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 564.992849] LustreError: 27023:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 565.111188] Lustre: server umount lustre-MDT0000 complete [ 572.629268] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 573.924477] Key type lgssc unregistered [ 574.078629] LNet: 27855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.081492] LNetError: 27855:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.088748] LNet: Removed LNI 192.168.201.105@tcp [ 574.476522] Key type .llcrypt unregistered [ 574.478261] Key type ._llcrypt unregistered [ 582.847033] Key type ._llcrypt registered [ 582.848899] Key type .llcrypt registered [ 582.889262] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 583.213208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 583.224125] alg: No test for adler32 (adler32-zlib) [ 584.118591] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 584.214065] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 585.808211] Key type lgssc registered [ 586.216303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 589.032108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 589.041687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.224391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.649283] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 594.720533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 594.845646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 596.156484] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 597.902637] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 599.086409] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 601.220760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 601.331305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 603.367829] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 605.437974] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 614.631723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 617.940419] LustreError: 31300:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 623.011194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.011859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 623.015041] Lustre: Skipped 1 previous similar message [ 627.120194] LustreError: 31553:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 627.185429] Lustre: server umount lustre-OST0000 complete [ 630.752947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.755771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.759287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.760931] Lustre: Skipped 1 previous similar message [ 633.313225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.313687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.751524] LustreError: 31754:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 634.753141] LustreError: 31754:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 634.781291] Lustre: server umount lustre-MDT0000 complete [ 636.052578] LustreError: 29065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791541 with bad export cookie 8600214738026402563 [ 636.056261] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.056869] LustreError: 29065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 636.117182] LustreError: 31957:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 636.119776] LustreError: 31957:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 636.196628] Lustre: server umount lustre-MDT0001 complete [ 639.718177] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 640.802711] Key type lgssc unregistered [ 640.943463] LNet: 32489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.945315] LNetError: 32489:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.952361] LNet: Removed LNI 192.168.201.105@tcp [ 641.291234] Key type .llcrypt unregistered [ 641.292546] Key type ._llcrypt unregistered [ 648.620955] Key type ._llcrypt registered [ 648.622448] Key type .llcrypt registered [ 648.660298] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 648.961756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.980932] alg: No test for adler32 (adler32-zlib) [ 649.838319] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 649.919730] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 651.504112] Key type lgssc registered [ 651.869795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.378430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 654.383695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.526097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.723697] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 659.496756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 659.616036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 660.855163] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 662.684174] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 663.884469] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 666.870806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 667.059403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.424607] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 674.369420] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 680.375764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 690.658681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 690.659069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.671795] LustreError: Skipped 1 previous similar message [ 690.683472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 690.687632] Lustre: Skipped 1 previous similar message [ 693.330452] LustreError: 35983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 693.655548] Lustre: server umount lustre-OST0000 complete [ 700.897243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 700.899854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.927539] Lustre: Skipped 1 previous similar message [ 700.933734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.940961] Lustre: Skipped 1 previous similar message [ 706.019038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.021617] Lustre: Skipped 1 previous similar message [ 711.149557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.162848] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 712.400127] LustreError: 36186:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 712.408558] LustreError: 36186:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 712.633569] Lustre: server umount lustre-MDT0000 complete [ 716.262773] LustreError: 33713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.297735] LustreError: 33713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 717.941902] LustreError: 33698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791623 with bad export cookie 17636634737339775917 [ 717.948768] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 717.971589] LustreError: 33698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 718.120502] LustreError: 36390:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 718.123399] LustreError: 36390:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 718.566860] Lustre: server umount lustre-MDT0001 complete [ 725.592859] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 727.175855] Key type lgssc unregistered [ 727.325522] LNet: 36921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.328869] LNetError: 36921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.342933] LNet: Removed LNI 192.168.201.105@tcp [ 727.761966] Key type .llcrypt unregistered [ 727.763493] Key type ._llcrypt unregistered [ 739.016170] Key type ._llcrypt registered [ 739.017430] Key type .llcrypt registered [ 739.057372] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 739.413971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 739.461032] alg: No test for adler32 (adler32-zlib) [ 740.336252] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 740.442471] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 742.064210] Key type lgssc registered [ 742.510627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 745.880105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 745.890044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.124765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 748.733112] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 752.376417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 752.528329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 754.161392] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 756.350512] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 757.871762] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 761.015858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 761.219705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 763.990625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 766.564100] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 771.507882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 781.800882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.809552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 781.844778] Lustre: Skipped 2 previous similar messages [ 781.867709] Lustre: Skipped 2 previous similar messages [ 784.308206] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 786.121893] LustreError: 40570:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 786.584196] Lustre: server umount lustre-MDT0000 complete [ 786.914088] LustreError: 38310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 786.942094] LustreError: 38310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 789.390815] LustreError: 38307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.172739] LustreError: 38290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791696 with bad export cookie 14344496643418984317 [ 791.176953] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 791.191540] LustreError: 38290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 792.038179] LustreError: 38307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 792.041765] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.089195] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 794.522449] LustreError: 38307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 794.559855] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 797.427255] LustreError: 40772:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 797.429917] LustreError: 40772:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 797.748204] Lustre: server umount lustre-MDT0001 complete [ 814.103921] LustreError: 40997:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 814.119229] LustreError: 40997:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 814.283321] Lustre: server umount lustre-OST0000 complete [ 835.482642] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 838.964978] Key type lgssc unregistered [ 839.207754] LNet: 41834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.214503] LNetError: 41834:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.236405] LNet: Removed LNI 192.168.201.105@tcp [ 839.991389] Key type .llcrypt unregistered [ 839.993909] Key type ._llcrypt unregistered [ 861.210178] Key type ._llcrypt registered [ 861.222769] Key type .llcrypt registered [ 861.396970] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 863.466194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 863.836722] alg: No test for adler32 (adler32-zlib) [ 865.044937] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 865.607353] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 867.440193] Key type lgssc registered [ 869.490830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 878.958483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 878.985475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.619163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.703691] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 890.858257] LustreError: 43245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 890.876840] LustreError: 43245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 892.369795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 892.654934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 896.470038] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 901.812353] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 904.755896] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 910.850588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 911.120866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.535721] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 921.395815] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 925.494944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 932.303250] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 932.835618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.838520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.866884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 935.906431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 935.909152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 935.924797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.989416] Lustre: Skipped 1 previous similar message [ 937.226125] LustreError: 45494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 937.582197] Lustre: server umount lustre-OST0000 complete [ 956.390586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.399975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 956.426467] Lustre: Skipped 1 previous similar message [ 956.462061] Lustre: Skipped 1 previous similar message [ 960.546500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 960.548960] Lustre: Skipped 1 previous similar message [ 969.184346] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 969.266702] LustreError: 45868:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 969.272325] LustreError: 45868:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 969.349876] Lustre: server umount lustre-MDT0000 complete [ 970.723214] LustreError: 43856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.738584] LustreError: 43856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 972.482726] LustreError: 43226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764791878 with bad export cookie 5686867649431658952 [ 972.485248] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.489163] LustreError: 43226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 978.711901] LustreError: 46073:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 978.720907] LustreError: 46073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 978.916943] Lustre: server umount lustre-MDT0001 complete [ 988.110668] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 990.066381] Key type lgssc unregistered [ 990.307995] LNet: 46604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.321704] LNetError: 46604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.350330] LNet: Removed LNI 192.168.201.105@tcp [ 991.028845] Key type .llcrypt unregistered [ 991.033721] Key type ._llcrypt unregistered [ 1008.045440] Key type ._llcrypt registered [ 1008.046865] Key type .llcrypt registered [ 1008.098157] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1008.694188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.754914] alg: No test for adler32 (adler32-zlib) [ 1009.731554] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 1009.915149] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1011.544174] Key type lgssc registered [ 1012.488362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1019.426135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1019.439284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.805983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.867209] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1031.158336] LustreError: 47982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.165218] LustreError: 47982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1035.215454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1035.694541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.658111] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1050.548357] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1055.304154] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1063.383914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1063.792586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.853630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1071.610218] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1079.675541] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1087.123214] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1091.206783] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1104.720680] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-12-0315h00m06s [ 1106.983849] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1112.040042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.046550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.110348] Lustre: Skipped 1 previous similar message [ 1112.115855] Lustre: Skipped 2 previous similar messages [ 1113.195399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.209698] Lustre: Skipped 3 previous similar messages [ 1116.297764] LustreError: 50552:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1116.428466] Lustre: server umount lustre-MDT0000 complete [ 1119.632448] LustreError: 48301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.531833] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1125.768525] LustreError: 47977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.783349] LustreError: 47977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1127.426975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.598298] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.964546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.056590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1129.015818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1132.069514] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1136.062814] Lustre: DEBUG MARKER: Wait for df (35977) ... [ 1137.735297] Lustre: DEBUG MARKER: done [ 1142.756429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.760097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.765486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.778976] Lustre: Skipped 2 previous similar messages [ 1143.271674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1148.195199] LustreError: 51607:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1148.208259] LustreError: 51607:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1148.540651] Lustre: server umount lustre-OST0000 complete [ 1154.529635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.533598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1154.570173] Lustre: Skipped 1 previous similar message [ 1154.576664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.592478] Lustre: Skipped 4 previous similar messages [ 1159.287170] LustreError: 51810:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1159.289867] LustreError: 51810:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1159.469658] Lustre: server umount lustre-MDT0000 complete [ 1161.699723] LustreError: 47982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.744592] LustreError: 47982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1163.016371] LustreError: 47961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792068 with bad export cookie 12758712646438236406 [ 1163.025629] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.133811] LustreError: 52024:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1163.137299] LustreError: 52024:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1163.335662] Lustre: server umount lustre-MDT0001 complete [ 1174.029425] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1177.233826] Key type lgssc unregistered [ 1177.548662] LNet: 52556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.561355] LNetError: 52556:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.587574] LNet: Removed LNI 192.168.201.105@tcp [ 1178.419658] Key type .llcrypt unregistered [ 1178.422384] Key type ._llcrypt unregistered [ 1212.992579] Key type ._llcrypt registered [ 1212.997457] Key type .llcrypt registered [ 1213.123765] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1232.154789] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1233.591222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1233.632050] alg: No test for adler32 (adler32-zlib) [ 1234.686092] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 1234.944177] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1236.656238] Key type lgssc registered [ 1237.840810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1246.687308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1253.295218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1259.527710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1266.172994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1272.789200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1272.846437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1272.861969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.055142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1274.084267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1274.165756] Lustre: lustre-MDT0000: new disk, initializing [ 1274.247973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.269115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1277.770088] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1287.285105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1287.355391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1287.425178] Lustre: 56512:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1287.525347] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1287.540290] Lustre: Skipped 1 previous similar message [ 1287.636614] Lustre: lustre-MDT0001: new disk, initializing [ 1287.763636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1287.810288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1287.816068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1291.347772] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1298.543617] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1301.768919] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1308.635934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1308.697558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1308.928800] Lustre: lustre-OST0000: new disk, initializing [ 1308.936379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1309.047367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.375366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1314.391150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1314.540150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1314.776323] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1323.027469] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1327.495441] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1327.746925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1331.779560] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1332.002764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1335.265546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1335.266786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.269123] LustreError: Skipped 1 previous similar message [ 1335.278071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1335.283200] Lustre: Skipped 1 previous similar message [ 1339.757462] LustreError: 59451:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1339.953518] Lustre: server umount lustre-OST0000 complete [ 1345.508614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.512793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1345.513613] Lustre: Skipped 2 previous similar messages [ 1345.533363] Lustre: Skipped 1 previous similar message [ 1350.346338] LustreError: 59654:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1350.349324] LustreError: 59654:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1350.513976] Lustre: server umount lustre-MDT0000 complete [ 1355.502437] LustreError: 56504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792261 with bad export cookie 17947184718345734179 [ 1355.508112] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1355.688893] LustreError: 59856:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1355.699914] LustreError: 59856:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1356.086700] Lustre: server umount lustre-MDT0001 complete [ 1365.247616] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 15:04:29 (1764792269) [ 1371.212573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1371.684877] LustreError: 60528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.705321] LustreError: 60528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1371.747406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.400769] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1377.250072] LustreError: 60528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.355232] LustreError: 60528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.248173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1383.592817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1387.620961] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1394.386596] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1399.730892] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1409.072036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1409.610853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.717897] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1420.379757] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1437.119815] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-12-0315h05m40s [ 1439.229790] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1448.363655] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1451.461120] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1451.541734] LustreError: 62688:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1471.980207] Lustre: lustre-MDT0000: Client e3d1b243-89ea-485c-8357-68a88ef17358 (at 192.168.201.5@tcp) reconnecting [ 1491.564126] LustreError: 62688:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1494.950526] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1499.551356] Lustre: DEBUG MARKER: done [ 1508.833566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1508.862011] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.906065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1510.882301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.921684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.978844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1510.981762] Lustre: Skipped 1 previous similar message [ 1514.311112] LustreError: 63558:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1514.323320] LustreError: 63558:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1514.579427] Lustre: server umount lustre-OST0000 complete [ 1521.122785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.128642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.156524] Lustre: Skipped 1 previous similar message [ 1521.174325] Lustre: Skipped 1 previous similar message [ 1525.166506] LustreError: 63759:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1525.184061] LustreError: 63759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1525.321715] Lustre: server umount lustre-MDT0000 complete [ 1528.294341] LustreError: 61241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1528.317493] LustreError: 61241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1529.303713] LustreError: 60507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792435 with bad export cookie 17947184718345735061 [ 1529.307577] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1529.323205] LustreError: 60507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1529.676443] Lustre: server umount lustre-MDT0001 complete [ 1538.971638] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1541.353925] Key type lgssc unregistered [ 1541.664139] LNet: 64497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.678061] LNetError: 64497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.703709] LNet: Removed LNI 192.168.201.105@tcp [ 1542.384841] Key type .llcrypt unregistered [ 1542.387630] Key type ._llcrypt unregistered [ 1573.562422] Key type ._llcrypt registered [ 1573.569647] Key type .llcrypt registered [ 1573.757663] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1590.378533] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1591.949512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1591.983191] alg: No test for adler32 (adler32-zlib) [ 1593.116403] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 1593.312826] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1594.944198] Key type lgssc registered [ 1596.702997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1605.814637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1612.382801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1618.670571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1625.587561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1632.545560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1632.605956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1632.619461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.887661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1633.921641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1633.971514] Lustre: lustre-MDT0000: new disk, initializing [ 1634.102692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.131250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1637.713837] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1647.407024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1647.455537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1647.580530] Lustre: 68449:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1647.634367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1647.637041] Lustre: Skipped 1 previous similar message [ 1647.679511] Lustre: lustre-MDT0001: new disk, initializing [ 1647.751513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1647.796177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1647.804890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1651.781979] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1659.771915] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1663.202952] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1669.125107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1669.225621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1669.514767] Lustre: lustre-OST0000: new disk, initializing [ 1669.520204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1669.637337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.346096] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1679.409552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1679.418695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1679.521782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1682.167968] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1685.734279] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1685.974573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1689.301046] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1689.441573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1695.202917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.205466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1695.255954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1695.258226] Lustre: Skipped 1 previous similar message [ 1696.989777] LustreError: 71387:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1697.305176] Lustre: server umount lustre-OST0000 complete [ 1704.417696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1704.426182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.432768] Lustre: Skipped 1 previous similar message [ 1704.437269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1705.441413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1705.444586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1707.868271] LustreError: 71589:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1707.904445] LustreError: 71589:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1708.211544] Lustre: server umount lustre-MDT0000 complete [ 1713.404526] LustreError: 68439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792619 with bad export cookie 745124489465839177 [ 1713.425180] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1713.443410] LustreError: 68439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1713.645822] LustreError: 71790:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1713.651063] LustreError: 71790:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1714.005889] Lustre: server umount lustre-MDT0001 complete [ 1723.453991] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 15:10:27 (1764792627) [ 1724.926509] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1726.726727] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 15:10:30 (1764792630) [ 1737.554372] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1742.811900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.310938] LustreError: 72840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.349059] LustreError: 72840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1743.382912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.361600] Lustre: Failing over lustre-MDT0000 [ 1744.368908] LustreError: 72868:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1744.388630] LustreError: 72868:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1744.394611] LustreError: 72868:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1744.482732] LustreError: 72919:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1744.485614] LustreError: 72919:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1744.790548] Lustre: server umount lustre-MDT0000 complete [ 1752.187072] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:10:55 (1764792655) [ 1758.972417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.265193] LustreError: 73489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1759.316654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.128297] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1764.321880] LustreError: 73489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1769.450198] LustreError: 73489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.515859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1772.069571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1776.942350] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1783.602713] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1787.739698] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1793.626241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1793.996623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.906189] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1806.090502] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1811.510841] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1817.656449] LustreError: 75832:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1817.668236] LustreError: 75832:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1817.824373] Lustre: server umount lustre-MDT0000 complete [ 1818.082395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1818.086100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.111742] LustreError: 74202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1821.547613] LustreError: 73470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792727 with bad export cookie 745124489465840332 [ 1821.553162] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1821.996366] Lustre: server umount lustre-MDT0001 complete [ 1826.063524] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1830.915373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1839.423274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.928134] Lustre: 66657:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764792730/real 1764792730] req@00000000972a3734 x1850519059504128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764792746 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1840.951692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.956501] Lustre: Skipped 2 previous similar messages [ 1843.168160] Lustre: 66660:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764792732/real 1764792732] req@000000006e7acb22 x1850519059504256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764792748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1845.280705] Lustre: 66660:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764792735/real 1764792735] req@0000000044712985 x1850519059504384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764792751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1847.143742] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa5736c67c0e4a3c [ 1847.150187] Lustre: MGC192.168.201.105@tcp: Connection restored to 0@lo (at 0@lo) [ 1847.499639] LustreError: 75097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.519832] LustreError: 75097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1847.596275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.666428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1851.080169] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1852.705233] Lustre: 66658:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764792742/real 1764792742] req@00000000df0dcceb x1850519059504768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764792758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1852.726254] Lustre: 66658:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1860.077840] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1860.497634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1866.732781] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1867.176327] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1875.436873] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1880.790776] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1891.813062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1891.820502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.862661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1891.868525] Lustre: Skipped 1 previous similar message [ 1895.388277] LustreError: 78299:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1895.399575] LustreError: 78299:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1895.481197] Lustre: server umount lustre-MDT0000 complete [ 1897.444330] LustreError: 76893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.463503] LustreError: 76893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 1898.634417] LustreError: 73470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792804 with bad export cookie 745124489465842236 [ 1898.639205] LustreError: 73470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1898.641035] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.063736] Lustre: server umount lustre-MDT0001 complete [ 1904.232297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1911.804718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.944500] Lustre: 66657:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764792808/real 1764792808] req@00000000f2530d9e x1850519059532160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764792824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1918.971367] Lustre: 66657:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1918.977969] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.985575] Lustre: Skipped 2 previous similar messages [ 1924.139418] LustreError: 66656:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000274e1291 x1850519059533312/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1924.674640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.679827] Lustre: Skipped 1 previous similar message [ 1924.739625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1928.951413] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1929.704693] LustreError: 77615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1929.721220] LustreError: 77615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1930.725871] Lustre: 66657:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764792819/real 1764792819] req@000000005c65a5e9 x1850519059532800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764792835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1930.783435] Lustre: 66657:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1938.327796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1942.084696] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1944.480373] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1948.206696] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1950.692863] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1953.920366] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1961.821070] LustreError: 80757:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1961.823323] LustreError: 80757:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1961.960105] Lustre: server umount lustre-MDT0000 complete [ 1963.491794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.645737] LustreError: 75834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764792871 with bad export cookie 745124489465843608 [ 1965.650844] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1965.660837] LustreError: 75834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1966.064145] Lustre: server umount lustre-MDT0001 complete [ 1972.842987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1976.253289] Lustre: DEBUG MARKER: files compared the same [ 1988.584489] Lustre: server umount lustre-OST0000 complete [ 2007.311848] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2009.382730] Key type lgssc unregistered [ 2009.707778] LNet: 82390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.712593] LNetError: 82390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.731209] LNet: Removed LNI 192.168.201.105@tcp [ 2010.471631] Key type .llcrypt unregistered [ 2010.473881] Key type ._llcrypt unregistered [ 2028.920629] Key type ._llcrypt registered [ 2028.922192] Key type .llcrypt registered [ 2028.999117] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2029.696335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2029.730187] alg: No test for adler32 (adler32-zlib) [ 2030.668070] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2030.838725] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2032.496181] Key type lgssc registered [ 2033.337801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.659824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.684482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.243617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.598689] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 2051.557178] LustreError: 83767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.579642] LustreError: 83767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2055.754281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2056.109719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2060.243559] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 2065.471622] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2069.173947] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2076.820744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2077.303874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.291263] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug +malloc all [ 2086.520539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2088.511979] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2096.106031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2096.109853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.115129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2097.122995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2097.125840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.133401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2102.081970] LustreError: 86014:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2102.240191] Lustre: server umount lustre-OST0000 complete [ 2107.361566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.377927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2107.383521] Lustre: Skipped 2 previous similar messages [ 2111.822699] LustreError: 86217:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2111.825322] LustreError: 86217:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2111.907958] Lustre: server umount lustre-MDT0000 complete [ 2112.487670] LustreError: 83767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.505984] LustreError: 83767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2115.569953] LustreError: 83746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793021 with bad export cookie 16732360688243599533 [ 2115.576038] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2115.596073] LustreError: 83746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2115.780917] LustreError: 86418:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2115.784638] LustreError: 86418:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2116.160565] Lustre: server umount lustre-MDT0001 complete [ 2127.998294] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2131.156121] Key type lgssc unregistered [ 2131.536923] LNet: 86951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.546399] LNetError: 86951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.566569] LNet: Removed LNI 192.168.201.105@tcp [ 2132.425034] Key type .llcrypt unregistered [ 2132.428118] Key type ._llcrypt unregistered [ 2155.509270] Key type ._llcrypt registered [ 2155.510940] Key type .llcrypt registered [ 2155.652520] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2156.658439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.700597] alg: No test for adler32 (adler32-zlib) [ 2157.814737] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2158.048909] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2159.736240] Key type lgssc registered [ 2161.107856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2172.972977] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2172.997399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2188.576381] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2193.699071] LustreError: 88321:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 2220.448196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.845084] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2235.796481] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2244.598764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.728419] Lustre: *** cfs_fail_loc=706, val=0*** [ 2245.100217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.167134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2248.017343] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2250.219041] LustreError: 88347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.236638] LustreError: 88347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2254.309900] LustreError: 89295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.336371] LustreError: 89295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2254.392995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2254.642592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2257.956101] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2262.323818] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2266.891860] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2270.689520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2270.693534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.711803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.297199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2275.300962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.304156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2276.405620] LustreError: 90648:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2276.925715] Lustre: server umount lustre-OST0000 complete [ 2285.537475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2285.542358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.544531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.544540] Lustre: Skipped 1 previous similar message [ 2285.632328] Lustre: Skipped 1 previous similar message [ 2290.658399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.661325] Lustre: Skipped 1 previous similar message [ 2295.777181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.780382] Lustre: Skipped 1 previous similar message [ 2297.312560] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2297.515262] LustreError: 90850:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2297.532591] LustreError: 90850:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2297.635054] Lustre: server umount lustre-MDT0000 complete [ 2301.921389] LustreError: 89298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.948191] LustreError: 89298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2303.990513] LustreError: 88328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793209 with bad export cookie 7160133599561883170 [ 2304.012168] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.027371] LustreError: 88328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2304.514561] LustreError: 91053:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2304.541392] LustreError: 91053:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2305.236835] Lustre: server umount lustre-MDT0001 complete [ 2323.698692] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2327.971098] Key type lgssc unregistered [ 2328.342235] LNet: 91587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.344773] LNetError: 91587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.363056] LNet: Removed LNI 192.168.201.105@tcp [ 2329.529929] Key type .llcrypt unregistered [ 2329.538323] Key type ._llcrypt unregistered [ 2343.563416] Key type ._llcrypt registered [ 2343.564405] Key type .llcrypt registered [ 2343.611201] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2343.926513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.934127] alg: No test for adler32 (adler32-zlib) [ 2344.804665] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2344.901157] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2346.488240] Key type lgssc registered [ 2346.930068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2350.037399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2350.045368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.752843] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2355.566301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2355.762185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2357.142075] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2360.460199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2360.585780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.500740] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2365.307780] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.311994] Lustre: MGS: Client 7b0cd688-ccd7-4add-b42e-5a333d847924 (at 0@lo) reconnecting [ 2365.395966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.422680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2366.894314] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2375.649317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2375.652374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.656198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2376.675176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2376.675455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.675985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2380.786085] LustreError: 95595:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2380.852864] Lustre: server umount lustre-OST0000 complete [ 2386.913891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.913919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2386.914558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2386.914562] Lustre: Skipped 1 previous similar message [ 2386.918297] Lustre: Skipped 1 previous similar message [ 2388.346688] LustreError: 95798:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2388.348270] LustreError: 95798:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2388.356925] Lustre: server umount lustre-MDT0000 complete [ 2389.654768] LustreError: 96000:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2389.656851] LustreError: 96000:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2389.730497] Lustre: server umount lustre-MDT0001 complete [ 2390.996945] Lustre: server umount MGS complete [ 2394.820122] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:21:39 (1764793299) [ 2404.769188] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 2407.920089] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2411.318544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2413.445942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2415.409489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2417.601352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2419.982280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2420.005594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.345056] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2425.312972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2425.342290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2425.425120] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2425.441912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2425.477119] Lustre: lustre-MDT0001: new disk, initializing [ 2425.503694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2425.511825] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2425.513980] Lustre: 100709:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2425.516227] LustreError: 100709:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2426.849822] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2431.001359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2431.027779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2431.145684] Lustre: lustre-OST0000: new disk, initializing [ 2431.148181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2431.172895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.263783] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2436.532113] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2438.972432] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2438.976944] Lustre: MGS: Client fa2a45ce-1174-4d47-90f1-fa620dd97b99 (at 0@lo) reconnecting [ 2438.992694] Lustre: 99996:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2439.000766] LustreError: 99996:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2439.015431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2439.054475] Lustre: lustre-MDT0000: new disk, initializing [ 2439.126877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2440.033716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2440.038967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2440.229644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2440.521671] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2459.617287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2459.620208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.625345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2459.627936] Lustre: Skipped 1 previous similar message [ 2465.329084] LustreError: 102942:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2465.331766] LustreError: 102942:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2465.398751] Lustre: server umount lustre-OST0000 complete [ 2470.882645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2470.882887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.885195] LustreError: Skipped 1 previous similar message [ 2470.889389] Lustre: Skipped 1 previous similar message [ 2470.891813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2470.894730] Lustre: Skipped 3 previous similar messages [ 2481.120160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2481.122426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2481.126852] Lustre: Skipped 3 previous similar messages [ 2481.159471] LustreError: 103143:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 2481.163086] LustreError: 103143:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2481.174888] Lustre: server umount lustre-MDT0000 complete [ 2488.665972] Lustre: server umount lustre-MDT0001 complete [ 2490.137681] LustreError: 103549:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2490.140582] LustreError: 103549:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2490.169388] Lustre: server umount MGS complete [ 2494.467816] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:23:19 (1764793399) [ 2501.302461] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2502.462811] Key type lgssc unregistered [ 2502.607448] LNet: 104883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2502.611315] LNetError: 104883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.618453] LNet: Removed LNI 192.168.201.105@tcp [ 2502.956116] Key type .llcrypt unregistered [ 2502.957164] Key type ._llcrypt unregistered [ 2510.329481] Key type ._llcrypt registered [ 2510.330722] Key type .llcrypt registered [ 2510.374943] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2510.806740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2510.870776] alg: No test for adler32 (adler32-zlib) [ 2511.753334] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2511.852681] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2513.440213] Key type lgssc registered [ 2513.855581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2516.662062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2516.668746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.814139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.135503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2522.019603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2522.144641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2523.447265] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2525.303265] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2526.548859] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2528.970464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2529.104953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.248960] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2533.295195] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2542.445563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2547.683789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2547.683987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.686850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2547.687173] LustreError: Skipped 1 previous similar message [ 2547.694707] Lustre: Skipped 1 previous similar message [ 2548.718634] LustreError: 108329:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2548.787927] Lustre: server umount lustre-OST0000 complete [ 2552.256769] LustreError: 108530:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2552.259658] LustreError: 108530:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2552.286095] Lustre: server umount lustre-MDT0000 complete [ 2552.802594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2552.802690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.805888] LustreError: 106063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.811215] Lustre: Skipped 1 previous similar message [ 2552.817018] LustreError: 106063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2553.683331] LustreError: 106783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793459 with bad export cookie 2668691633574590048 [ 2553.686344] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.687566] LustreError: 106783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2553.760843] LustreError: 108732:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2553.763147] LustreError: 108732:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2553.849416] Lustre: server umount lustre-MDT0001 complete [ 2557.779825] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2558.911931] Key type lgssc unregistered [ 2559.054660] LNet: 109263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2559.058040] LNetError: 109263:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2559.064322] LNet: Removed LNI 192.168.201.105@tcp [ 2559.402693] Key type .llcrypt unregistered [ 2559.404171] Key type ._llcrypt unregistered [ 2566.371502] Key type ._llcrypt registered [ 2566.373037] Key type .llcrypt registered [ 2566.417033] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2566.767232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2566.833317] alg: No test for adler32 (adler32-zlib) [ 2567.700076] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2567.787942] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2569.376162] Key type lgssc registered [ 2569.796115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2572.138282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2572.142100] LustreError: 110301:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2572.148638] LustreError: 110305:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2572.152282] LustreError: 110301:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2572.156617] LustreError: 110301:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2572.160630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.161643] LustreError: 110301:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2573.398691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.443685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2576.583775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2578.108957] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2580.114736] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing lsmod [ 2581.266269] LustreError: 111481:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2581.266572] LustreError: 111480:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2581.272563] LustreError: 111481:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2581.276322] LustreError: 111481:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2581.280946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2581.282289] LustreError: 111481:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2581.438389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.593814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.595229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.599474] Lustre: Skipped 2 previous similar messages [ 2586.601348] Lustre: Skipped 1 previous similar message [ 2587.617924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.620190] Lustre: Skipped 2 previous similar messages [ 2588.671517] LustreError: 111671:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2588.702602] Lustre: server umount lustre-MDT0000 complete [ 2590.173460] LustreError: 110324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793495 with bad export cookie 2030435393018294644 [ 2590.175356] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2590.178285] LustreError: 110324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2591.712261] LustreError: 110777:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d2d9ef19 x1850520080760192/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2591.725353] LustreError: 111875:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2591.727557] LustreError: 111875:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2591.814957] Lustre: server umount lustre-MDT0001 complete [ 2602.485989] LustreError: 112076:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2602.488318] LustreError: 112076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2602.584706] Lustre: server umount lustre-OST0000 complete [ 2606.065694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.245375] LustreError: 112553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.254051] LustreError: 112553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2606.275912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.728903] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2610.817744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2610.955619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2612.388818] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2614.307728] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2615.625547] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2618.066901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2620.329164] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2622.417522] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2630.507418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2635.746587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.747319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2635.748176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2635.751031] Lustre: Skipped 1 previous similar message [ 2636.591058] LustreError: 114821:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2636.593132] LustreError: 114821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2636.655692] Lustre: server umount lustre-OST0000 complete [ 2640.087538] Lustre: server umount lustre-MDT0000 complete [ 2640.865513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.865985] LustreError: 112552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.869926] Lustre: Skipped 1 previous similar message [ 2640.876494] LustreError: 112552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2641.389559] LustreError: 112532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793547 with bad export cookie 2030435393018295316 [ 2641.392325] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.393132] LustreError: 112532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2641.456787] LustreError: 115222:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2641.459137] LustreError: 115222:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2641.542206] Lustre: server umount lustre-MDT0001 complete [ 2645.267687] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2646.397235] Key type lgssc unregistered [ 2646.533354] LNet: 115752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.537900] LNetError: 115752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.546469] LNet: Removed LNI 192.168.201.105@tcp [ 2646.878235] Key type .llcrypt unregistered [ 2646.879512] Key type ._llcrypt unregistered [ 2656.265172] Key type ._llcrypt registered [ 2656.266759] Key type .llcrypt registered [ 2656.300733] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2656.742241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2656.755554] alg: No test for adler32 (adler32-zlib) [ 2657.606364] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2657.686566] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2659.264290] Key type lgssc registered [ 2659.637666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2662.366956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2662.375875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.548414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.966081] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2668.205375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2668.344521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2669.833159] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2671.861376] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2673.179770] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2675.588963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2675.709906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.692645] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2679.668462] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2689.068396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2693.420836] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2694.181162] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2699.237526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2699.239893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.240995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.248056] Lustre: Skipped 1 previous similar message [ 2701.230652] LustreError: 119492:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2701.295286] Lustre: server umount lustre-OST0000 complete [ 2704.353689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2704.354208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.356017] LustreError: Skipped 1 previous similar message [ 2704.360829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.363530] Lustre: Skipped 2 previous similar messages [ 2708.734549] LustreError: 119698:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2708.737421] LustreError: 119698:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2708.766494] Lustre: server umount lustre-MDT0000 complete [ 2709.473517] LustreError: 117840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.478380] LustreError: 117840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2710.084125] LustreError: 119345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793615 with bad export cookie 16869333881886928406 [ 2710.087790] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2710.088052] LustreError: 119345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2710.159205] LustreError: 119899:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2710.161810] LustreError: 119899:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2710.260356] Lustre: server umount lustre-MDT0001 complete [ 2713.966831] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2715.071890] Key type lgssc unregistered [ 2715.204484] LNet: 120430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2715.208249] LNetError: 120430:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2715.217545] LNet: Removed LNI 192.168.201.105@tcp [ 2715.544643] Key type .llcrypt unregistered [ 2715.545825] Key type ._llcrypt unregistered [ 2722.540889] Key type ._llcrypt registered [ 2722.542076] Key type .llcrypt registered [ 2722.582635] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2723.096982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2723.129743] alg: No test for adler32 (adler32-zlib) [ 2723.988659] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2724.073233] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2725.656117] Key type lgssc registered [ 2726.032640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2728.772768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2728.778573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.950087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.319547] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2734.384774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2734.513736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2735.956286] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2738.106420] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2739.519757] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2742.027287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2742.156387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2744.213132] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2746.153214] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2754.471285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2759.649622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.651056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.656884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.659292] Lustre: Skipped 1 previous similar message [ 2761.265097] LustreError: 123878:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2761.333921] Lustre: server umount lustre-OST0000 complete [ 2764.733643] LustreError: 124081:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2764.735547] LustreError: 124081:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2764.765579] Lustre: server umount lustre-MDT0000 complete [ 2764.769916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.771067] LustreError: 121606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.775460] Lustre: Skipped 2 previous similar messages [ 2764.780278] LustreError: 121606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2766.043202] LustreError: 121589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793671 with bad export cookie 15716560607724790397 [ 2766.045991] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2766.048156] LustreError: 121589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2766.101135] LustreError: 124282:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2766.103314] LustreError: 124282:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2766.181582] Lustre: server umount lustre-MDT0001 complete [ 2769.934676] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2770.998742] Key type lgssc unregistered [ 2771.134407] LNet: 124812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2771.137082] LNetError: 124812:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.146363] LNet: Removed LNI 192.168.201.105@tcp [ 2771.451371] Key type .llcrypt unregistered [ 2771.452914] Key type ._llcrypt unregistered [ 2781.123037] Key type ._llcrypt registered [ 2781.124656] Key type .llcrypt registered [ 2781.172627] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2781.670759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2781.679981] alg: No test for adler32 (adler32-zlib) [ 2782.541463] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2782.630645] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2784.216193] Key type lgssc registered [ 2784.633179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.529307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2787.535855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.723675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.302779] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2793.443983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2793.574145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2794.957668] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2796.915150] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2798.203975] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2800.555345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2800.672716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.684185] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2804.690674] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2813.988078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2817.835080] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2819.055536] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2819.057791] Lustre: Skipped 2 previous similar messages [ 2820.469060] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2820.471292] Lustre: Skipped 2 previous similar messages [ 2826.749717] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2826.763643] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2829.608952] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2829.612219] Lustre: Skipped 2 previous similar messages [ 2834.436808] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.5@tcp 192.168.0.[2,10]@tcp [ 2837.570166] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.5/24@tcp in log lustre-MDT0000 [ 2837.572686] Lustre: Skipped 2 previous similar messages [ 2839.559421] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.5/24@tcp [ 2839.562192] Lustre: Skipped 1 previous similar message [ 2844.641158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.643552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.644225] LustreError: Skipped 1 previous similar message [ 2844.649300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.652464] Lustre: Skipped 1 previous similar message [ 2849.762163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.764275] Lustre: Skipped 1 previous similar message [ 2850.095098] LustreError: 129999:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2850.164253] Lustre: server umount lustre-OST0000 complete [ 2854.882187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.882283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.882709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.885944] Lustre: Skipped 2 previous similar messages [ 2857.661889] LustreError: 130200:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2857.664365] LustreError: 130200:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2857.700408] Lustre: server umount lustre-MDT0000 complete [ 2859.081668] LustreError: 126166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793764 with bad export cookie 7253877779313750522 [ 2859.084848] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2859.086137] LustreError: 126166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2859.149072] LustreError: 130402:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2859.151434] LustreError: 130402:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2859.247425] Lustre: server umount lustre-MDT0001 complete [ 2863.232769] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2864.292412] Key type lgssc unregistered [ 2864.408333] LNet: 130933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2864.411355] LNetError: 130933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2864.420288] LNet: Removed LNI 192.168.201.105@tcp [ 2864.720949] Key type .llcrypt unregistered [ 2864.722780] Key type ._llcrypt unregistered [ 2875.184324] Key type ._llcrypt registered [ 2875.185277] Key type .llcrypt registered [ 2875.221179] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2875.624051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2875.637938] alg: No test for adler32 (adler32-zlib) [ 2876.493833] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2876.587796] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2878.168162] Key type lgssc registered [ 2878.560876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2881.268192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2881.274199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.439903] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2882.445098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.449768] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.5/24@tcp [ 2883.900743] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2886.903862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2887.005471] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2887.014589] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.5/24@tcp [ 2887.030236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.301180] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2890.192964] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2891.504295] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2893.958294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2894.090854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.108209] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2898.016228] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2908.466765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2912.737919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2912.741512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.745687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.761379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2913.763136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.763823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.769893] Lustre: Skipped 1 previous similar message [ 2917.741026] LustreError: 134771:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2917.798716] Lustre: server umount lustre-OST0000 complete [ 2922.976762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2922.979429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.983996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2925.114582] LustreError: 134973:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2925.116456] LustreError: 134973:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2925.143910] Lustre: server umount lustre-MDT0000 complete [ 2926.413246] LustreError: 134073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793832 with bad export cookie 3064103329902839747 [ 2926.416022] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.416601] LustreError: 134073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2926.478996] LustreError: 135174:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2926.482279] LustreError: 135174:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2926.585061] Lustre: server umount lustre-MDT0001 complete [ 2930.320155] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 2931.542045] Key type lgssc unregistered [ 2931.677589] LNet: 135705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2931.680347] LNetError: 135705:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2931.690498] LNet: Removed LNI 192.168.201.105@tcp [ 2932.012442] Key type .llcrypt unregistered [ 2932.013461] Key type ._llcrypt unregistered [ 2941.883545] Key type ._llcrypt registered [ 2941.885291] Key type .llcrypt registered [ 2941.932907] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 2948.276366] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2948.841734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2948.860747] alg: No test for adler32 (adler32-zlib) [ 2949.744557] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 2949.840026] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 2951.424180] Key type lgssc registered [ 2951.838452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2954.966171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2957.140928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2959.313056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2961.482786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2964.008897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2964.035118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2964.043060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.155495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2965.171541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2965.216315] Lustre: lustre-MDT0000: new disk, initializing [ 2965.250755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.258505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2966.692433] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2971.095602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2971.119423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2971.154690] Lustre: 139053:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2971.182937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2971.185385] Lustre: Skipped 1 previous similar message [ 2971.227686] Lustre: lustre-MDT0001: new disk, initializing [ 2971.272736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2971.290253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2971.294479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2972.703435] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2975.987984] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2977.309874] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2979.707025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2979.733399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2979.853667] Lustre: lustre-OST0000: new disk, initializing [ 2979.856586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2979.891952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.272666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2980.276794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2980.310937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2981.833304] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2985.092524] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2986.590550] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2986.664121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2988.058616] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2988.148860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2990.563361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2990.563739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.566904] LustreError: Skipped 1 previous similar message [ 2990.573441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2990.576871] Lustre: Skipped 1 previous similar message [ 2995.683055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.685022] Lustre: Skipped 1 previous similar message [ 3000.803737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.806903] Lustre: Skipped 1 previous similar message [ 3003.360159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3003.384172] LustreError: 141993:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3003.456944] Lustre: server umount lustre-OST0000 complete [ 3005.922728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.923428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.926316] Lustre: Skipped 2 previous similar messages [ 3011.041321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.043435] Lustre: Skipped 2 previous similar messages [ 3019.232150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3019.262499] LustreError: 142196:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3019.264827] LustreError: 142196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3019.289367] Lustre: server umount lustre-MDT0000 complete [ 3020.556478] LustreError: 139833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793926 with bad export cookie 12075149017100030876 [ 3020.560068] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.562069] LustreError: 139833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3020.623634] LustreError: 142400:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3020.625816] LustreError: 142400:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3020.722658] Lustre: server umount lustre-MDT0001 complete [ 3023.547958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.761161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.123802] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3027.920207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3028.041698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3029.331619] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3031.104278] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3032.250108] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3034.457327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3036.477338] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3038.342569] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3039.781158] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3049.076904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3050.412330] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3050.490507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3053.742033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.765224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3053.817534] Lustre: lustre-OST0001: new disk, initializing [ 3053.819680] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3053.848843] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3053.851598] Lustre: Skipped 1 previous similar message [ 3055.736395] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3059.056543] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3061.762655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3061.766579] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3061.793078] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3062.211992] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3063.623624] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3063.715107] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3064.922177] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99bfc54d0000.ost_server_uuid 50 [ 3065.431344] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99bfc54d0000.ost_server_uuid in FULL state after 0 sec [ 3071.971821] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3071.974396] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.976088] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3071.977291] Lustre: Skipped 1 previous similar message [ 3071.979706] Lustre: Skipped 2 previous similar messages [ 3073.327063] LustreError: 147208:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3073.328935] LustreError: 147208:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3073.352035] Lustre: server umount lustre-OST0001 complete [ 3077.090989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.092064] LustreError: 144472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.094020] Lustre: Skipped 1 previous similar message [ 3077.099397] LustreError: 144472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3080.621068] LustreError: 147409:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3080.622726] LustreError: 147409:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3080.689105] Lustre: server umount lustre-OST0000 complete [ 3082.209450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.157161] Lustre: server umount lustre-MDT0000 complete [ 3089.446101] LustreError: 144494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764793995 with bad export cookie 12075149017100031744 [ 3089.448939] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.449156] LustreError: 144494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3089.520409] LustreError: 147812:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3089.522603] LustreError: 147812:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3089.630110] Lustre: server umount lustre-MDT0001 complete [ 3096.520623] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3097.710862] Key type lgssc unregistered [ 3097.850519] LNet: 148800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3097.854452] LNetError: 148800:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3097.862460] LNet: Removed LNI 192.168.201.105@tcp [ 3098.202924] Key type .llcrypt unregistered [ 3098.204249] Key type ._llcrypt unregistered [ 3110.105936] Key type ._llcrypt registered [ 3110.107034] Key type .llcrypt registered [ 3110.147849] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3116.856940] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3117.450516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3117.462751] alg: No test for adler32 (adler32-zlib) [ 3118.331467] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 3118.434346] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3120.016222] Key type lgssc registered [ 3120.436407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3123.556034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3125.889911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3128.157371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3130.586075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.064711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.088237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3133.094898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.207688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3134.224481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3134.272161] Lustre: lustre-MDT0000: new disk, initializing [ 3134.303169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.312172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3135.744515] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3140.128150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.150786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3140.188297] Lustre: 152721:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3140.215886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3140.219471] Lustre: Skipped 1 previous similar message [ 3140.257994] Lustre: lustre-MDT0001: new disk, initializing [ 3140.297979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3140.314140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3140.319264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3141.743434] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3144.981395] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3146.251034] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3148.653431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3148.679695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3148.786921] Lustre: lustre-OST0000: new disk, initializing [ 3148.790340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3148.820848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.856227] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3152.444741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3152.449382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3152.486993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3154.179829] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3155.676830] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3155.772553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3157.202310] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3157.292724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3159.984077] LustreError: 155662:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3160.063967] Lustre: server umount lustre-OST0000 complete [ 3162.595364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.596251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3162.601510] Lustre: Skipped 1 previous similar message [ 3167.713467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3167.716037] Lustre: Skipped 2 previous similar messages [ 3171.813469] LustreError: 155862:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3171.816338] LustreError: 155862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3171.849558] Lustre: server umount lustre-MDT0000 complete [ 3173.225207] LustreError: 152714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794078 with bad export cookie 6506964330020615064 [ 3173.229189] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3182.565826] LustreError: 156064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3182.568252] LustreError: 156064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3182.662417] Lustre: server umount lustre-MDT0001 complete [ 3185.545304] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 15:34:50 (1764794090) [ 3190.941890] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3193.985021] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3197.487987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.875640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3202.160283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3204.353083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3206.700141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3206.718296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3206.817935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3206.831805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3206.871498] Lustre: lustre-MDT0000: new disk, initializing [ 3206.896887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3206.904266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3208.237802] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3213.428464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.471490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3213.530458] Lustre: 159049:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3213.565878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3213.568957] Lustre: Skipped 1 previous similar message [ 3213.621898] Lustre: lustre-MDT0001: new disk, initializing [ 3213.682700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3213.703511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3213.709266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3215.889752] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3221.149693] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3223.343467] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3227.872322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3227.930817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3228.244448] Lustre: lustre-OST0000: new disk, initializing [ 3228.247115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3228.314529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.272517] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3236.408313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3236.419450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3236.614318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3238.741252] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3242.399752] Lustre: Failing over lustre-OST0000 [ 3242.462110] LustreError: 161492:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3242.469537] LustreError: 161492:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3242.655584] Lustre: server umount lustre-OST0000 complete [ 3257.378751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3257.531449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.539599] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3259.917145] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3260.640167] Lustre: 150937:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764794150/real 1764794150] req@00000000420aef4f x1850520658570240/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1764794166 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3260.650099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.660158] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3260.845448] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3260.845613] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3261.621912] Lustre: Failing over lustre-MDT0000 [ 3261.698579] LustreError: 162467:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3261.700912] LustreError: 162467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3261.748103] Lustre: server umount lustre-MDT0000 complete [ 3264.992997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3264.996401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.000661] Lustre: Skipped 1 previous similar message [ 3265.002866] LustreError: 159060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.008593] LustreError: 159060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3265.938501] LustreError: 159687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.064213] Lustre: 150938:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764794157/real 1764794157] req@00000000e69799f0 x1850520658572160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764794173 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3271.051141] LustreError: 161236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.059338] LustreError: 161236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3274.208256] Lustre: 150938:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764794163/real 1764794163] req@000000000bc767b4 x1850520658573312/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764794179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3275.507180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.555699] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3275.560458] Lustre: 159052:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.105@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3275.678643] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3276.171103] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3277.256691] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3280.870434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3280.872891] Lustre: Skipped 1 previous similar message [ 3280.881725] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3280.899621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3285.987301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3285.988626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3285.992698] Lustre: Skipped 3 previous similar messages [ 3285.997603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3288.184419] LustreError: 163351:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3288.187108] LustreError: 163351:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3288.268115] Lustre: server umount lustre-OST0000 complete [ 3291.106625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3291.106773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.115951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.118503] Lustre: Skipped 2 previous similar messages [ 3295.963225] Lustre: server umount lustre-MDT0000 complete [ 3296.225127] LustreError: 159060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.231752] LustreError: 159060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3297.617250] LustreError: 159831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794203 with bad export cookie 6506964330020617136 [ 3297.622186] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.624473] LustreError: 159831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3297.825641] Lustre: server umount lustre-MDT0001 complete [ 3302.565588] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3303.906664] Key type lgssc unregistered [ 3304.060780] LNet: 164285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3304.065253] LNetError: 164285:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3304.076579] LNet: Removed LNI 192.168.201.105@tcp [ 3304.449156] Key type .llcrypt unregistered [ 3304.450893] Key type ._llcrypt unregistered [ 3326.787214] Key type ._llcrypt registered [ 3326.788747] Key type .llcrypt registered [ 3326.835228] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3327.279761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3327.338275] alg: No test for adler32 (adler32-zlib) [ 3328.225329] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 3328.335752] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3329.928341] Key type lgssc registered [ 3330.348542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3333.336636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3333.346215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.530769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.832148] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3339.136417] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3341.339880] LustreError: 165811:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3341.343922] LustreError: 165811:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3341.348800] LustreError: 165811:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 3341.419583] LustreError: 166334:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3341.535487] Lustre: server umount lustre-MDT0000 complete [ 3347.004907] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3350.462316] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3354.915821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.668631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3363.353295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3371.208805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3378.420408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3378.501763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.857958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3378.892383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3378.967468] Lustre: lustre-MDT0000: new disk, initializing [ 3379.086148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.114947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3383.282741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3394.511966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3394.570970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3394.655215] Lustre: 168980:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3394.717790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3394.722095] Lustre: Skipped 1 previous similar message [ 3394.816486] Lustre: lustre-MDT0001: new disk, initializing [ 3394.924309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3394.973152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3394.985697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3399.939874] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3408.454031] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3412.553736] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3417.907193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3417.948700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3418.107469] Lustre: lustre-OST0000: new disk, initializing [ 3418.117258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3418.163678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.321617] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3424.285361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3424.291193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3424.352117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3425.681672] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3428.275877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3429.397573] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:38:54 (1764794334) [ 3526.625200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.625747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.630644] Lustre: Skipped 2 previous similar messages [ 3526.633311] Lustre: Skipped 1 previous similar message [ 3530.563779] LustreError: 172484:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3530.566607] LustreError: 172484:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3530.618947] Lustre: server umount lustre-MDT0000 complete [ 3531.745669] LustreError: 168985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.752640] LustreError: 168985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3532.069236] LustreError: 168971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794437 with bad export cookie 13861549729597630282 [ 3532.075124] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3536.864496] LustreError: 168985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.864528] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.873653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3537.888524] LustreError: 168986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.888538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3538.176502] LustreError: 172688:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3538.179375] LustreError: 172688:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3538.256706] Lustre: server umount lustre-MDT0001 complete [ 3545.769858] LustreError: 172890:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3545.772693] LustreError: 172890:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3545.836641] Lustre: server umount lustre-OST0000 complete [ 3548.842300] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3551.839958] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3555.007331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3557.086195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3559.119100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3561.235792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3563.512542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3563.533455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.606312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3563.616381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3563.648140] Lustre: lustre-MDT0000: new disk, initializing [ 3563.669562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.675216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3564.926642] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3568.918796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.939607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3568.971096] Lustre: 175223:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3568.994053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3568.995529] Lustre: Skipped 1 previous similar message [ 3569.028271] Lustre: lustre-MDT0001: new disk, initializing [ 3569.068509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3569.071216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3570.322406] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3573.483738] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3574.711100] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3576.923984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3576.944968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3577.035855] Lustre: lustre-OST0000: new disk, initializing [ 3577.037888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3577.062572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.065105] Lustre: Skipped 1 previous similar message [ 3578.680344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3578.684380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3578.720117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3578.931593] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3581.933460] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3583.207683] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3583.278160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3584.558616] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3584.632279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3587.310654] LustreError: 178160:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3587.312207] LustreError: 178160:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3587.363222] Lustre: server umount lustre-OST0000 complete [ 3589.090681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.091259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3589.095780] Lustre: Skipped 1 previous similar message [ 3589.099469] Lustre: Skipped 1 previous similar message [ 3594.209322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.211789] Lustre: Skipped 1 previous similar message [ 3598.308156] LustreError: 178361:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3598.310171] LustreError: 178361:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3598.336805] Lustre: server umount lustre-MDT0000 complete [ 3599.329166] LustreError: 176094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.333800] LustreError: 176094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3599.638212] LustreError: 175214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794505 with bad export cookie 13861549729597789112 [ 3599.640861] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3599.642641] LustreError: 175214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3610.177135] Lustre: server umount lustre-MDT0001 complete [ 3612.992269] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:41:57 (1764794517) [ 3618.029571] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3621.441204] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3624.813758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3627.008759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3629.074149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3631.246374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.612564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.635928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.717857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.729639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3633.768256] Lustre: lustre-MDT0000: new disk, initializing [ 3633.794504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.804061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3635.135074] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3639.425239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.454862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3639.494694] Lustre: 181551:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3639.606495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3640.957338] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3644.109812] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3645.394899] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3647.711137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3647.737070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3648.757808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3648.796815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3649.751513] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3652.856464] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3659.237433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3659.237651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.240263] LustreError: Skipped 1 previous similar message [ 3659.244404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3659.246903] Lustre: Skipped 1 previous similar message [ 3660.910092] LustreError: 184099:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3660.912146] LustreError: 184099:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3660.989393] Lustre: server umount lustre-OST0000 complete [ 3664.354871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.358976] Lustre: Skipped 2 previous similar messages [ 3668.445988] Lustre: server umount lustre-MDT0000 complete [ 3669.474663] LustreError: 181562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3669.479447] LustreError: 181562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3669.729355] LustreError: 181543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794575 with bad export cookie 13861549729597789980 [ 3669.732436] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3669.732808] LustreError: 181543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3669.887163] Lustre: server umount lustre-MDT0001 complete [ 3672.847490] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:42:57 (1764794577) [ 3678.176776] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3681.302599] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3684.624695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3686.893156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3689.002642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3691.198221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3693.509798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3693.537078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.638287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3693.652472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3693.655552] Lustre: Skipped 3 previous similar messages [ 3693.695534] Lustre: lustre-MDT0000: new disk, initializing [ 3693.696958] Lustre: Skipped 2 previous similar messages [ 3693.725579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.728029] Lustre: Skipped 2 previous similar messages [ 3693.734552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3693.739083] Lustre: Skipped 2 previous similar messages [ 3695.132371] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3699.279541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3699.302059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3699.335407] Lustre: 187486:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3699.369295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3699.371179] Lustre: Skipped 2 previous similar messages [ 3699.447585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3700.689604] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3703.875863] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3705.125331] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3707.443557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3707.469937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3707.873678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3709.706835] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3712.933397] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3718.115614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3718.115938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.830093] LustreError: 190032:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3720.841758] LustreError: 190032:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3720.965658] Lustre: server umount lustre-OST0000 complete [ 3725.281683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3725.287626] LustreError: Skipped 1 previous similar message [ 3730.303055] Lustre: server umount lustre-MDT0000 complete [ 3732.704417] LustreError: 187477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794638 with bad export cookie 13861549729597791051 [ 3732.705792] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3732.715767] LustreError: 187477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3733.040090] Lustre: server umount lustre-MDT0001 complete [ 3739.870819] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3741.636961] Key type lgssc unregistered [ 3741.851878] LNet: 190970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3741.859098] LNetError: 190970:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3741.871313] LNet: Removed LNI 192.168.201.105@tcp [ 3742.339168] Key type .llcrypt unregistered [ 3742.341974] Key type ._llcrypt unregistered [ 3758.645582] Key type ._llcrypt registered [ 3758.649258] Key type .llcrypt registered [ 3758.749940] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3759.583468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3759.707849] alg: No test for adler32 (adler32-zlib) [ 3760.719807] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 3760.898667] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3762.576364] Key type lgssc registered [ 3763.437239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3768.994424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3769.007075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.741443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.254835] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3781.090354] LustreError: 192346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.166738] LustreError: 192346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3782.116043] LustreError: 192346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.055975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3787.234527] LustreError: 192346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.623533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3791.955770] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3796.767178] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3799.245143] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3802.951308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3803.146755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.296511] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3809.162350] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3816.490260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3832.802062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.807830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.815456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.902391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3836.902965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.905801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.924601] Lustre: Skipped 1 previous similar message [ 3838.527136] LustreError: 194623:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3838.644711] Lustre: server umount lustre-OST0000 complete [ 3842.018848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.020111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3842.026190] Lustre: Skipped 1 previous similar message [ 3847.140910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3847.151436] Lustre: Skipped 2 previous similar messages [ 3847.388958] LustreError: 194824:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3847.393086] LustreError: 194824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3847.461240] Lustre: server umount lustre-MDT0000 complete [ 3849.561324] LustreError: 192329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794755 with bad export cookie 7451925214826684755 [ 3849.565625] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.567883] LustreError: 192329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3849.663020] LustreError: 195025:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3849.668513] LustreError: 195025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3849.814609] Lustre: server umount lustre-MDT0001 complete [ 3856.115497] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3858.277605] Key type lgssc unregistered [ 3858.484161] LNet: 195557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3858.489838] LNetError: 195557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3858.498529] LNet: Removed LNI 192.168.201.105@tcp [ 3859.224394] Key type .llcrypt unregistered [ 3859.227199] Key type ._llcrypt unregistered [ 3894.540361] Key type ._llcrypt registered [ 3894.542250] Key type .llcrypt registered [ 3894.716236] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3896.516310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3896.609598] alg: No test for adler32 (adler32-zlib) [ 3897.933797] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 3898.189640] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3899.888189] Key type lgssc registered [ 3901.540877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3915.926876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3915.955925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.114141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.399719] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3927.526681] LustreError: 196955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.552411] LustreError: 196955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3932.648135] LustreError: 196955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.787655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3934.168833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3938.080901] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3942.782432] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3945.802624] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3952.191201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3952.609647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.459927] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3962.681306] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3962.875689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3970.492860] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 3973.094596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.099975] Lustre: Skipped 1 previous similar message [ 3973.100265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.101991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.109242] Lustre: Skipped 1 previous similar message [ 3975.242867] LustreError: 199202:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3975.378291] Lustre: server umount lustre-OST0000 complete [ 3980.779905] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99bfc6453000.ost_server_uuid 50 [ 3982.268788] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3997.665819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.681784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.687588] Lustre: Skipped 1 previous similar message [ 4001.953964] LustreError: 199570:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4001.959895] LustreError: 199570:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4002.006495] Lustre: server umount lustre-MDT0000 complete [ 4002.791305] LustreError: 196956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4002.820922] LustreError: 196956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4005.572380] LustreError: 198576:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764794911 with bad export cookie 12174143562066500937 [ 4005.575596] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.586399] LustreError: 198576:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4005.797578] LustreError: 199771:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4005.805740] LustreError: 199771:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4006.046179] Lustre: server umount lustre-MDT0001 complete [ 4014.277442] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:48:38 (1764794918) [ 4021.092358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4021.552599] LustreError: 200444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.634843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.656213] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4026.873679] LustreError: 200444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4033.266181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4033.704878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.433295] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4042.473646] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4046.142529] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4051.576321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4051.848343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4056.397676] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4060.621717] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4065.168847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4065.233540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4065.402908] Lustre: lustre-OST0001: new disk, initializing [ 4065.413425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4067.565656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4068.586297] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4071.960286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4071.964688] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4072.029156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4072.599711] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4074.465534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4074.469662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.475598] Lustre: Skipped 1 previous similar message [ 4074.478845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4077.034218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.147576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.150497] Lustre: Skipped 4 previous similar messages [ 4088.288413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4088.315160] LustreError: 203661:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4088.320319] LustreError: 203661:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4088.345144] Lustre: server umount lustre-OST0000 complete [ 4091.251068] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4091.363529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4092.392422] LustreError: 202053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.401297] LustreError: 202053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4093.194677] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4093.293187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4107.748491] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4107.748576] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4107.751682] LustreError: Skipped 1 previous similar message [ 4107.758865] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4107.764138] Lustre: Skipped 5 previous similar messages [ 4109.942454] LustreError: 204366:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4109.947525] LustreError: 204366:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4110.052379] Lustre: server umount lustre-OST0001 complete [ 4118.216141] LustreError: 204567:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4118.219124] LustreError: 204567:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4118.271908] Lustre: server umount lustre-MDT0000 complete [ 4120.112053] LustreError: 200424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795025 with bad export cookie 12174143562066502015 [ 4120.117106] LustreError: 200424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4120.117727] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4120.298218] Lustre: server umount lustre-MDT0001 complete [ 4122.874117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4124.753768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4127.066105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4129.522038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4137.136772] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:50:41 (1764795041) [ 4145.499306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.096360] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4146.133694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4146.799675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4146.802339] Lustre: Skipped 1 previous similar message [ 4152.342274] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4163.129836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4163.273647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4163.348466] Lustre: 206259:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4168.895741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4177.841969] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4183.001485] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4190.988318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4191.243830] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4191.451565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.457981] Lustre: Skipped 1 previous similar message [ 4196.575449] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4201.090882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4201.493829] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4206.813350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4206.883499] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4211.780953] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4216.480164] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4219.372306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4219.377175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4219.378295] Lustre: Skipped 4 previous similar messages [ 4219.386157] Lustre: Skipped 5 previous similar messages [ 4224.699211] LustreError: 209458:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4224.708591] LustreError: 209458:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4224.758977] Lustre: server umount lustre-OST0000 complete [ 4227.474520] LustreError: 207891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.486517] LustreError: 207891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4244.554596] Lustre: server umount lustre-OST0001 complete [ 4250.094517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.137184] Lustre: Skipped 1 previous similar message [ 4255.201372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4255.203701] Lustre: Skipped 5 previous similar messages [ 4259.302832] LustreError: 209885:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4259.343873] LustreError: 209885:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4259.483794] Lustre: server umount lustre-MDT0000 complete [ 4260.320207] Lustre: 196266:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764795150/real 1764795150] req@00000000df0dcceb x1850521475495424/t0(0) o400->lustre-OST0001-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764795166 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4260.321111] LustreError: 208954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.397754] LustreError: 208954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4261.282714] Lustre: 196268:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764795150/real 1764795150] req@00000000224b7e4f x1850521475495680/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1764795166 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4264.005641] LustreError: 207909:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795169 with bad export cookie 12174143562066503422 [ 4264.014446] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4264.028714] LustreError: 207909:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4264.844353] Lustre: server umount lustre-MDT0001 complete [ 4273.109532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4277.474227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4281.971520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4285.574068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4292.072319] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:53:16 (1764795196) [ 4306.717503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 4316.302991] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4324.903202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4330.325646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4336.483509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.986795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4349.631874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4349.697387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.898617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4349.937713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4349.988814] Lustre: lustre-MDT0000: new disk, initializing [ 4350.059474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4350.062535] Lustre: Skipped 1 previous similar message [ 4350.094797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4354.119946] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4364.459376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.529290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4364.657320] Lustre: 213915:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4364.741128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4364.753682] Lustre: Skipped 1 previous similar message [ 4364.842890] Lustre: lustre-MDT0001: new disk, initializing [ 4365.044399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4365.056993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4368.702132] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4375.969671] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4379.800499] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4387.465584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4387.539219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4387.730885] Lustre: lustre-OST0000: new disk, initializing [ 4387.740416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4394.535351] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4397.643933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4397.657213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4397.773324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4402.058818] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4406.686378] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4406.986668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4411.110207] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4411.299353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4413.414411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.415905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4413.417249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4413.428377] Lustre: Skipped 2 previous similar messages [ 4413.430517] Lustre: Skipped 1 previous similar message [ 4427.232170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4427.314032] LustreError: 216857:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4427.317679] LustreError: 216857:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4427.491793] Lustre: server umount lustre-OST0000 complete [ 4431.845296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4431.848361] LustreError: Skipped 1 previous similar message [ 4437.336397] Lustre: server umount lustre-MDT0000 complete [ 4439.011216] LustreError: 215091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4439.032279] LustreError: 215091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4441.461545] LustreError: 213908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795347 with bad export cookie 12174143562066504983 [ 4441.464199] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4441.487820] LustreError: 213908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4442.042365] Lustre: server umount lustre-MDT0001 complete [ 4451.114124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.239346] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4465.261798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4470.464698] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4476.675542] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4480.825356] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4487.921715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4488.358641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.373450] Lustre: Skipped 4 previous similar messages [ 4495.045105] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4501.556651] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4506.333384] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4506.597129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4510.971281] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4511.233369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4516.325958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4516.339120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.357211] Lustre: Skipped 2 previous similar messages [ 4519.526489] Lustre: server umount lustre-OST0000 complete [ 4525.678853] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4531.133879] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4560.782888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4567.481278] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4567.501310] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4567.875608] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4574.339383] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4579.673765] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 4582.913777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.941172] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4588.006151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4588.012495] LustreError: Skipped 2 previous similar messages [ 4588.013237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.025053] Lustre: Skipped 11 previous similar messages [ 4589.881797] LustreError: 222122:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4589.892686] LustreError: 222122:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4590.059996] Lustre: server umount lustre-OST0000 complete [ 4600.282448] Lustre: server umount lustre-MDT0000 complete [ 4603.365105] LustreError: 218720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4603.379103] LustreError: 218720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4603.618774] LustreError: 217716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795509 with bad export cookie 12174143562066505858 [ 4603.623079] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4603.627465] LustreError: 217716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4603.907436] Lustre: server umount lustre-MDT0001 complete [ 4612.340287] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:58:35 (1764795515) [ 4618.534928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.524163] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4629.273389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4632.794232] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4639.854160] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4645.216560] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4656.314356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4667.126193] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4675.149753] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4679.075200] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4679.373753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4683.274115] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4683.463219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4692.637496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4692.769751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4692.926629] Lustre: lustre-OST0001: new disk, initializing [ 4692.930888] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4699.826718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4699.851761] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4700.027248] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4702.136121] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4710.750397] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4715.033655] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4715.308727] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4719.412293] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4719.703931] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4725.749272] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.753059] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4725.762814] Lustre: Skipped 5 previous similar messages [ 4725.770161] LustreError: Skipped 1 previous similar message [ 4728.270369] Lustre: server umount lustre-OST0001 complete [ 4733.363729] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4733.634782] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4737.178389] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4737.352325] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4765.644939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4765.864147] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4765.877239] Lustre: Skipped 5 previous similar messages [ 4771.320837] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4771.343683] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4771.345809] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4771.710193] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4777.890813] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4779.535788] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4787.453868] Lustre: server umount lustre-OST0001 complete [ 4809.644262] Lustre: server umount lustre-OST0000 complete [ 4813.795315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4813.805082] LustreError: Skipped 2 previous similar messages [ 4827.104151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4829.318032] Lustre: server umount lustre-MDT0000 complete [ 4832.553742] LustreError: 223176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795738 with bad export cookie 12174143562066506824 [ 4832.562026] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4838.987970] Lustre: server umount lustre-MDT0001 complete [ 4843.601618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4846.601638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4849.739516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4852.933987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4858.603375] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:02:42 (1764795762) [ 4863.879893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.143438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4864.167816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4867.615963] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4874.923789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4875.004946] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4875.033269] Lustre: 231200:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4877.777978] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4882.269666] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4885.416754] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4891.239740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4891.420957] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4896.286047] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4901.129286] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4917.563365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4917.631414] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4923.317228] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4928.681899] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4930.648238] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4934.186697] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4934.376690] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4937.614328] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4937.861190] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4941.511740] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99bfc5ed0800.ost_server_uuid 50 [ 4942.871624] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99bfc5ed0800.ost_server_uuid in FULL state after 0 sec [ 4943.956763] Lustre: Permanently deactivating lustre-OST0001 [ 4943.976399] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4947.698342] Lustre: Permanently reactivating lustre-OST0001 [ 4956.132047] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4956.154426] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4956.157338] Lustre: Skipped 20 previous similar messages [ 4966.370581] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4966.405141] LustreError: 235188:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4966.415181] LustreError: 235188:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 4966.474609] Lustre: server umount lustre-OST0001 complete [ 4966.883586] LustreError: 232823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4966.919054] LustreError: 232823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 4975.696034] Lustre: server umount lustre-OST0000 complete [ 4982.241265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.253948] Lustre: Skipped 11 previous similar messages [ 4985.546988] Lustre: server umount lustre-MDT0000 complete [ 4989.352462] LustreError: 231191:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764795895 with bad export cookie 12174143562066508119 [ 4989.354361] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4989.367651] LustreError: 231191:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4989.890540] Lustre: server umount lustre-MDT0001 complete [ 4996.626068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5000.973894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5005.809590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5011.225499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5021.202044] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:05:24 (1764795924) [ 5022.322180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5029.949482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.310927] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5030.333774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5030.337512] Lustre: Skipped 5 previous similar messages [ 5034.129578] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5043.326226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5043.408381] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5043.439486] Lustre: 237352:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5047.482832] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5053.193187] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5058.042326] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5065.558972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5065.665752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5065.876048] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5065.904519] Lustre: Permanently deactivating lustre-OST0000 [ 5065.922145] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5072.816270] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5075.974487] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5078.048834] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5084.596409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5084.709664] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5085.429990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5089.802549] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5094.586768] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5096.585745] Lustre: Permanently reactivating lustre-OST0000 [ 5096.606170] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5096.610450] Lustre: Skipped 2 previous similar messages [ 5103.150710] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5103.154742] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.162067] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5103.167686] Lustre: Skipped 1 previous similar message [ 5103.197336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.215232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5111.150524] Lustre: server umount lustre-OST0001 complete [ 5121.816728] Lustre: server umount lustre-OST0000 complete [ 5133.179568] Lustre: server umount lustre-MDT0000 complete [ 5136.421332] LustreError: 237342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796042 with bad export cookie 12174143562066509764 [ 5136.429506] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.444995] LustreError: 237342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5136.858703] Lustre: server umount lustre-MDT0001 complete [ 5146.051613] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5148.597955] Key type lgssc unregistered [ 5148.919765] LNet: 241772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5148.931632] LNetError: 241772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5148.945955] LNet: Removed LNI 192.168.201.105@tcp [ 5149.690295] Key type .llcrypt unregistered [ 5149.691753] Key type ._llcrypt unregistered [ 5167.574782] Key type ._llcrypt registered [ 5167.577132] Key type .llcrypt registered [ 5167.641597] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5168.149112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5168.171050] alg: No test for adler32 (adler32-zlib) [ 5169.122708] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 5169.299562] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5170.936211] Key type lgssc registered [ 5172.332668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5175.412442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5181.675691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5181.707619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.349488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.721292] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5192.682854] LustreError: 243202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.688501] LustreError: 243202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5193.710675] LustreError: 243202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5195.796766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5195.856071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5195.908938] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5195.953118] Lustre: 243190:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5195.969573] Lustre: Permanently deactivating lustre-MDT0001 [ 5195.991676] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5196.241374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5199.530979] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5202.467553] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5204.915981] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5208.137623] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5215.380337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5215.776185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.900448] LustreError: 244822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.929220] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5221.386130] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5225.955056] LustreError: 245112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5226.644846] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5228.018440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5233.499594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5233.682476] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5238.797974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5238.815365] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5238.906111] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5244.016469] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5247.689866] Lustre: Permanently reactivating lustre-MDT0001 [ 5247.711395] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5247.715820] Lustre: Skipped 2 previous similar messages [ 5251.816082] Lustre: lustre-MDT0001: Client 84d6fbd4-41c8-470d-8b05-30689ad178c0 (at 192.168.201.5@tcp) reconnecting [ 5254.219974] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5254.244354] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 5254.250401] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5254.258847] LustreError: 246385:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 5254.259357] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5254.265578] LustreError: 246385:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 5254.283632] LustreError: 246385:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -108 [ 5254.296616] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 5254.296616] [ 5254.303103] LustreError: 243195:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5254.782962] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5256.103653] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5260.172564] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5260.328543] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5263.964514] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5264.166702] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5265.164753] Lustre: 244276:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 5265.171087] Lustre: 244276:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5265.176150] Lustre: 244276:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 5265.183223] Lustre: 244276:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 5265.189707] Lustre: 244276:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5265.195590] Lustre: 244276:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5266.367444] Lustre: Permanently deactivating lustre-MDT0001 [ 5266.381733] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5266.385541] Lustre: Skipped 2 previous similar messages [ 5272.118370] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5273.909134] Lustre: 243199:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 506 < left 979, rollback = 2 [ 5273.914711] Lustre: 243199:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 5273.920705] Lustre: 243199:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 5273.924704] Lustre: 243199:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5273.938064] Lustre: 243199:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/979/0 [ 5273.945192] Lustre: 243199:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5273.949594] Lustre: 243199:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5273.953864] Lustre: 243199:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5273.958263] Lustre: 243199:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 8/130/4, delete: 0/0/0 [ 5273.963096] Lustre: 243199:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5273.967323] Lustre: 243199:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 5273.974019] Lustre: 243199:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5276.649304] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5276.655874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.665343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.714640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.720265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.728179] Lustre: Skipped 1 previous similar message [ 5279.736533] Lustre: Skipped 1 previous similar message [ 5282.892064] LustreError: 247273:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5283.028957] Lustre: server umount lustre-MDT0000 complete [ 5284.839483] LustreError: 244276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.865605] LustreError: 244276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5286.296302] LustreError: 246313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796192 with bad export cookie 17790470721454464860 [ 5286.298938] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.301978] LustreError: 246313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5286.431469] LustreError: 247475:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5286.441936] LustreError: 247475:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5286.616164] Lustre: server umount lustre-MDT0001 complete [ 5292.047747] LustreError: 247675:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5292.055531] LustreError: 247675:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5292.120378] Lustre: server umount lustre-OST0000 complete [ 5302.254500] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:10:06 (1764796206) [ 5311.940201] LustreError: 248680:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5311.945860] LustreError: 248680:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5312.140906] Lustre: server umount lustre-OST0001 complete [ 5318.452816] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5328.361967] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5338.208320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5343.924886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5349.800333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5355.394630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5360.750671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5360.797830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5360.970367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5360.976398] Lustre: Skipped 2 previous similar messages [ 5361.000786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5361.079820] Lustre: lustre-MDT0000: new disk, initializing [ 5361.187523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5361.230946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5364.528749] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5373.085074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5373.170781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5373.259241] Lustre: 250867:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5373.325625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5373.340026] Lustre: Skipped 1 previous similar message [ 5373.438225] Lustre: lustre-MDT0001: new disk, initializing [ 5373.562431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5373.622056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5373.630316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5376.734486] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5382.774951] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5385.780155] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5391.605128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5391.677346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5391.938087] Lustre: lustre-OST0000: new disk, initializing [ 5391.940719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5392.034316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5396.572356] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5401.642776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5401.653951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5401.771912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5402.448071] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5404.058565] LustreError: 250874:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5411.159341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5411.258871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5411.374516] Lustre: lustre-OST0001: new disk, initializing [ 5411.387329] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5414.163913] LustreError: 250874:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5414.200898] LustreError: 250874:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5417.064870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5417.073930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5417.076995] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5417.157861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5423.524686] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5424.209314] LustreError: 250874:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5425.029528] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5427.100777] Lustre: lustre-OST0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 5427.692641] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.693454] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.708742] Lustre: Skipped 1 previous similar message [ 5432.206634] Lustre: lustre-OST0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 5432.212259] Lustre: Skipped 2 previous similar messages [ 5437.339962] Lustre: lustre-OST0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 5437.355708] Lustre: Skipped 2 previous similar messages [ 5440.992172] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5441.042093] LustreError: 254359:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5441.063478] LustreError: 254359:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5441.162071] Lustre: server umount lustre-OST0001 complete [ 5442.450541] LustreError: 252598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.409022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5458.409692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.432868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5458.441619] Lustre: Skipped 2 previous similar messages [ 5458.451738] Lustre: Skipped 3 previous similar messages [ 5463.532142] LustreError: 252596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.552459] LustreError: 252596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5464.385601] LustreError: 254586:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5464.388714] LustreError: 254586:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5464.554369] Lustre: server umount lustre-OST0000 complete [ 5468.644348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.651917] Lustre: Skipped 1 previous similar message [ 5474.283890] Lustre: server umount lustre-MDT0000 complete [ 5477.269630] LustreError: 250859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796383 with bad export cookie 17790470721454467674 [ 5477.278758] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5477.279944] LustreError: 250859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5483.481613] LustreError: 254989:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5483.483719] LustreError: 254989:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5483.646741] Lustre: server umount lustre-MDT0001 complete [ 5492.890843] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5495.295906] Key type lgssc unregistered [ 5495.559259] LNet: 255523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5495.567640] LNetError: 255523:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5495.584619] LNet: Removed LNI 192.168.201.105@tcp [ 5496.405655] Key type .llcrypt unregistered [ 5496.414042] Key type ._llcrypt unregistered [ 5520.840296] Key type ._llcrypt registered [ 5520.846080] Key type .llcrypt registered [ 5520.993284] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5535.021795] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5535.742987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5535.784256] alg: No test for adler32 (adler32-zlib) [ 5536.811440] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 5537.005180] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5538.656237] Key type lgssc registered [ 5539.694621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5549.138831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5555.259737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5561.517420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5567.476604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5574.347450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5574.441952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5574.474149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.752481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5575.801799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5575.918697] Lustre: lustre-MDT0000: new disk, initializing [ 5576.016408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.029391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5580.161391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5591.217420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5591.275664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5591.351928] Lustre: 259478:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5591.406693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5591.409785] Lustre: Skipped 1 previous similar message [ 5591.471930] Lustre: lustre-MDT0001: new disk, initializing [ 5591.581543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5591.611536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5591.619900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5595.557172] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5602.732659] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5606.701869] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5612.786859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5612.837890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5613.085682] Lustre: lustre-OST0000: new disk, initializing [ 5613.107114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5613.201476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.148304] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5621.382494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5621.395684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5621.571311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5627.494640] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5632.827517] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5633.115425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5637.026247] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5637.255057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5642.212277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.212520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.241511] LustreError: Skipped 1 previous similar message [ 5642.249869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.313755] Lustre: Skipped 1 previous similar message [ 5645.113147] LustreError: 262421:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5645.365896] Lustre: server umount lustre-OST0000 complete [ 5652.454933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.466391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.469117] Lustre: Skipped 1 previous similar message [ 5655.280772] LustreError: 262622:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5655.289101] LustreError: 262622:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5655.381488] Lustre: server umount lustre-MDT0000 complete [ 5658.875559] LustreError: 261235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796564 with bad export cookie 12537091447483750077 [ 5658.881521] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5658.902930] LustreError: 261235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5659.050639] LustreError: 262824:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5659.068914] LustreError: 262824:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5659.375723] Lustre: server umount lustre-MDT0001 complete [ 5668.423313] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 16:16:12 (1764796572) [ 5674.762791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.139716] LustreError: 263495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.149830] LustreError: 263495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5675.249284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.181327] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5680.610441] LustreError: 263495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5684.708232] LustreError: 263495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.616499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5687.954886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5691.658198] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5698.009393] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5701.657949] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5708.889900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5709.215929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.103936] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5719.569819] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5773.795304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5773.798713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.806898] Lustre: Skipped 1 previous similar message [ 5773.813886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.843117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.847442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5775.855992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.955162] LustreError: 265819:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5776.958346] LustreError: 265819:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5777.096981] Lustre: server umount lustre-OST0000 complete [ 5780.196882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5788.416896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5788.672966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.311368] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5795.738172] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5795.817266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.830679] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5800.937548] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.949774] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5836.770776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5836.771173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.784645] LustreError: Skipped 1 previous similar message [ 5836.813036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.827419] Lustre: Skipped 2 previous similar messages [ 5838.414111] LustreError: 267483:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5838.419523] LustreError: 267483:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5838.619475] Lustre: server umount lustre-OST0000 complete [ 5846.497177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5846.501628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.535218] Lustre: Skipped 1 previous similar message [ 5846.538183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5848.223936] LustreError: 267686:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5848.227015] LustreError: 267686:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5848.445891] Lustre: server umount lustre-MDT0000 complete [ 5851.666234] LustreError: 264213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796757 with bad export cookie 12537091447483750959 [ 5851.669703] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5851.692316] LustreError: 264213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5852.140253] Lustre: server umount lustre-MDT0001 complete [ 5861.109569] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5863.568886] Key type lgssc unregistered [ 5863.820582] LNet: 268421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5863.826334] LNetError: 268421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5863.838038] LNet: Removed LNI 192.168.201.105@tcp [ 5864.540970] Key type .llcrypt unregistered [ 5864.543511] Key type ._llcrypt unregistered [ 5886.337293] Key type ._llcrypt registered [ 5886.339611] Key type .llcrypt registered [ 5886.474154] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5887.835628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5887.854652] alg: No test for adler32 (adler32-zlib) [ 5888.953084] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 5889.165516] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5890.832209] Key type lgssc registered [ 5891.897160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5898.581539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5898.608392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.112888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.445588] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5909.480206] LustreError: 269819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5909.530908] LustreError: 269819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5911.934146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5912.424751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5916.706798] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5923.075196] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5927.041380] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5933.409910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5933.699720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.437580] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5941.504360] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5945.007707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5965.793944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5965.796791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5965.811315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5965.814584] Lustre: Skipped 1 previous similar message [ 5965.823249] Lustre: Skipped 1 previous similar message [ 5970.813315] LustreError: 272836:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5970.943831] Lustre: server umount lustre-OST0000 complete [ 5973.987037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5973.990597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.997883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5976.034711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5976.036156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.808951] LustreError: 273038:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5979.817024] LustreError: 273038:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5979.902779] Lustre: server umount lustre-MDT0000 complete [ 5981.155633] LustreError: 270817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.164140] LustreError: 270817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5982.406637] LustreError: 273039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764796888 with bad export cookie 866150196656471075 [ 5982.410859] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5982.422873] LustreError: 273039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5982.562194] LustreError: 273240:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5982.566985] LustreError: 273240:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5982.711152] Lustre: server umount lustre-MDT0001 complete [ 5989.857503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5992.355703] Key type lgssc unregistered [ 5992.579512] LNet: 273770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5992.587961] LNetError: 273770:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5992.599221] LNet: Removed LNI 192.168.201.105@tcp [ 5993.259835] Key type .llcrypt unregistered [ 5993.261488] Key type ._llcrypt unregistered [ 6009.316759] Key type ._llcrypt registered [ 6009.318509] Key type .llcrypt registered [ 6009.426961] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6010.593406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6010.710426] alg: No test for adler32 (adler32-zlib) [ 6011.858513] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 6012.113499] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6013.816265] Key type lgssc registered [ 6015.014842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6022.432170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6022.449893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6023.874701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.634585] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6033.387209] LustreError: 274973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.415564] LustreError: 274973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6036.627694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6037.045837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6041.493674] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6047.378207] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6050.500795] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6055.785277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6056.051434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.629868] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6065.205676] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6068.343787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6078.953659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.965520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.984886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6084.074776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6084.083407] Lustre: Skipped 2 previous similar messages [ 6084.608549] LustreError: 277367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6084.789645] Lustre: server umount lustre-OST0000 complete [ 6089.185755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6089.212578] Lustre: Skipped 2 previous similar messages [ 6089.213362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6094.305250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6094.308649] Lustre: Skipped 2 previous similar messages [ 6094.701220] LustreError: 277569:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6094.704778] LustreError: 277569:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6094.829886] Lustre: server umount lustre-MDT0000 complete [ 6098.559052] LustreError: 274954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797004 with bad export cookie 11950822561709170981 [ 6098.559662] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.589610] LustreError: 274954:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6098.733834] LustreError: 277773:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6098.745100] LustreError: 277773:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6099.044678] Lustre: server umount lustre-MDT0001 complete [ 6110.244830] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6112.911364] Key type lgssc unregistered [ 6113.266932] LNet: 278305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6113.278128] LNetError: 278305:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6113.293472] LNet: Removed LNI 192.168.201.105@tcp [ 6113.968476] Key type .llcrypt unregistered [ 6113.970338] Key type ._llcrypt unregistered [ 6136.336871] Key type ._llcrypt registered [ 6136.351473] Key type .llcrypt registered [ 6136.467340] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6137.868342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6137.875900] alg: No test for adler32 (adler32-zlib) [ 6139.065365] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 6139.370486] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6141.088181] Key type lgssc registered [ 6142.291966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6149.638501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6149.682718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.143781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.646324] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6160.354811] LustreError: 279704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.369702] LustreError: 279704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6162.142704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6162.508163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6165.798674] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6170.401678] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6173.760706] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6180.148797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6180.428583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.237737] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6189.867075] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6195.838905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6223.847449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6223.859584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.881993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.917218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.921247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.923324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.942202] Lustre: Skipped 1 previous similar message [ 6228.634180] LustreError: 282745:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6228.822277] Lustre: server umount lustre-OST0000 complete [ 6232.033639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.040582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6237.156646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6237.159508] Lustre: Skipped 2 previous similar messages [ 6242.275142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6242.279527] Lustre: Skipped 1 previous similar message [ 6246.368173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6246.495156] LustreError: 282947:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6246.500021] LustreError: 282947:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6246.567164] Lustre: server umount lustre-MDT0000 complete [ 6247.405088] LustreError: 279698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.415048] LustreError: 279698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6249.823597] LustreError: 279685:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797155 with bad export cookie 14113051986819900333 [ 6249.844169] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6249.864504] LustreError: 279685:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6249.996908] LustreError: 283151:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6250.020526] LustreError: 283151:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6250.401614] Lustre: server umount lustre-MDT0001 complete [ 6260.794081] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6263.668796] Key type lgssc unregistered [ 6263.955447] LNet: 283683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6263.963175] LNetError: 283683:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6263.979344] LNet: Removed LNI 192.168.201.105@tcp [ 6264.833501] Key type .llcrypt unregistered [ 6264.834982] Key type ._llcrypt unregistered [ 6280.976433] Key type ._llcrypt registered [ 6280.978179] Key type .llcrypt registered [ 6281.126120] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6282.368430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6282.420879] alg: No test for adler32 (adler32-zlib) [ 6283.523278] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 6283.939696] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6285.664119] Key type lgssc registered [ 6287.452745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6295.878711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6295.912196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.581886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.694707] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6307.854874] LustreError: 284914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6307.916723] LustreError: 284914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6309.630788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6309.943961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6312.990410] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6317.699807] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6320.826707] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6326.870990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6327.209611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.873608] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6337.993148] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6342.574711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6352.870145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6352.870945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.872283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6352.884561] Lustre: Skipped 1 previous similar message [ 6356.928305] LustreError: 287314:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6357.124087] Lustre: server umount lustre-OST0000 complete [ 6361.060343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6361.069127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.075800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6361.078727] Lustre: Skipped 1 previous similar message [ 6363.105517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.108692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.121970] Lustre: Skipped 1 previous similar message [ 6368.226084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6368.233797] Lustre: Skipped 1 previous similar message [ 6372.323727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6372.329909] Lustre: Skipped 1 previous similar message [ 6374.369295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6374.464046] LustreError: 287515:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6374.475056] LustreError: 287515:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6374.587397] Lustre: server umount lustre-MDT0000 complete [ 6377.443494] LustreError: 284890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.455676] LustreError: 284890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6377.601588] LustreError: 284867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797283 with bad export cookie 14650122986278128456 [ 6377.613947] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6377.616132] LustreError: 284867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6377.744446] LustreError: 287719:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6377.750459] LustreError: 287719:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6378.052182] Lustre: server umount lustre-MDT0001 complete [ 6388.476701] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6391.458788] Key type lgssc unregistered [ 6391.712574] LNet: 288252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6391.723555] LNetError: 288252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6391.750216] LNet: Removed LNI 192.168.201.105@tcp [ 6392.501362] Key type .llcrypt unregistered [ 6392.503172] Key type ._llcrypt unregistered [ 6402.626473] Key type ._llcrypt registered [ 6402.627881] Key type .llcrypt registered [ 6402.714966] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6419.364656] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6433.768946] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6434.912246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6434.993315] alg: No test for adler32 (adler32-zlib) [ 6436.053083] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 6436.230459] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6437.912169] Key type lgssc registered [ 6438.991155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6447.095534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6453.230842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6459.022541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6464.920118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6470.887730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6470.952187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6470.977570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.226658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6472.263880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6472.356868] Lustre: lustre-MDT0000: new disk, initializing [ 6472.489803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6472.506621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6476.447899] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6486.609380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6486.686464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6486.758661] Lustre: 291765:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6486.851590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6486.886332] Lustre: Skipped 1 previous similar message [ 6486.930318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6487.013310] Lustre: lustre-MDT0001: new disk, initializing [ 6487.131901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6487.203090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6487.221095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6491.283220] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6498.123269] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6501.447411] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6507.814368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6507.906826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6508.227331] Lustre: lustre-OST0000: new disk, initializing [ 6508.230770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6508.334316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.909285] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6517.821507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6517.832708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6517.896088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6520.439421] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6523.813684] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6524.003549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6527.117121] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6527.304134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6533.605313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6533.605452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.612388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6533.622660] Lustre: Skipped 1 previous similar message [ 6538.726319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.737596] Lustre: Skipped 2 previous similar messages [ 6543.328376] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6543.358131] LustreError: 294702:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6543.521788] Lustre: server umount lustre-OST0000 complete [ 6548.449973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6548.456179] LustreError: Skipped 1 previous similar message [ 6548.459736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6548.480311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.772908] LustreError: 294907:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6553.781971] LustreError: 294907:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6553.889360] Lustre: server umount lustre-MDT0000 complete [ 6557.062377] LustreError: 291756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797462 with bad export cookie 8404395112213340238 [ 6557.068530] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6557.090936] LustreError: 291756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6557.200372] LustreError: 295108:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6557.204728] LustreError: 295108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6557.431533] Lustre: server umount lustre-MDT0001 complete [ 6565.082868] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:31:09 (1764797469) [ 6571.041710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.545739] LustreError: 295780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.560758] LustreError: 295780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6571.622717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6575.233600] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6576.610083] LustreError: 295780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6581.729778] LustreError: 295780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.983434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6584.331564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6588.250265] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6593.501365] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6596.857470] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6603.461389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6603.801488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6608.754337] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6613.461495] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6627.298318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.300664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6627.306294] Lustre: Skipped 1 previous similar message [ 6627.308579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6627.318810] Lustre: Skipped 2 previous similar messages [ 6630.463349] LustreError: 298029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6630.483028] LustreError: 298029:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6630.646822] Lustre: server umount lustre-OST0000 complete [ 6635.489224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6635.494330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6635.510620] Lustre: Skipped 1 previous similar message [ 6635.517212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6639.950701] LustreError: 298233:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6639.954877] LustreError: 298233:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6640.056233] Lustre: server umount lustre-MDT0000 complete [ 6642.661700] LustreError: 295780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.400787] LustreError: 295759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797549 with bad export cookie 8404395112213341113 [ 6643.407070] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6643.426265] LustreError: 295759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6643.791654] Lustre: server umount lustre-MDT0001 complete [ 6654.287935] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 6656.790492] Key type lgssc unregistered [ 6657.144597] LNet: 298966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6657.156161] LNetError: 298966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6658.214341] LNet: Removed LNI 192.168.201.105@tcp [ 6658.989109] Key type .llcrypt unregistered [ 6658.990851] Key type ._llcrypt unregistered [ 6682.690945] Key type ._llcrypt registered [ 6682.693121] Key type .llcrypt registered [ 6682.774365] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6693.751920] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6694.652946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6694.719737] alg: No test for adler32 (adler32-zlib) [ 6695.806823] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 6696.049633] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 6697.712299] Key type lgssc registered [ 6698.689766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6706.463383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6713.086689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6718.783397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6724.710161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6729.806257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6734.474131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6740.474274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6740.530416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6740.544195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.751896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6741.794968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6741.872983] Lustre: lustre-MDT0000: new disk, initializing [ 6741.992809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.009112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6745.459511] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6755.284232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6755.376298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6755.433465] Lustre: 302753:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6755.518396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6755.521247] Lustre: Skipped 1 previous similar message [ 6755.688294] Lustre: lustre-MDT0001: new disk, initializing [ 6755.799944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6755.839284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6755.860522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6759.739344] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6766.952628] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6770.306113] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6776.911345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6776.971561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6777.180421] Lustre: lustre-OST2710: new disk, initializing [ 6777.183818] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6777.278096] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6781.532068] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6785.584906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6785.589485] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6785.668327] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6787.171888] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6792.659346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6792.740972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6792.951587] Lustre: lustre-OST03e8: new disk, initializing [ 6792.959185] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6793.062597] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6798.620631] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6803.036266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6803.048414] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6803.148209] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6803.411219] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6808.949408] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6809.177764] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6813.095903] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6813.351415] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6817.353613] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6817.664096] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6821.422957] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6821.630494] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6827.491336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6827.496882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6827.509548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6829.036819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.037067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6829.044318] Lustre: Skipped 1 previous similar message [ 6829.047147] Lustre: Skipped 1 previous similar message [ 6831.069767] LustreError: 307098:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6831.147381] Lustre: server umount lustre-MDT0000 complete [ 6834.190799] LustreError: 304498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797739 with bad export cookie 11727150789346828312 [ 6834.192144] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6834.210257] LustreError: 304498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6834.338247] LustreError: 307300:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6834.341259] LustreError: 307300:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6834.527750] Lustre: server umount lustre-MDT0001 complete [ 6847.499350] LustreError: 307501:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6847.502247] LustreError: 307501:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6847.609579] Lustre: server umount lustre-OST2710 complete [ 6860.862466] LustreError: 307703:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6860.868886] LustreError: 307703:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6861.066065] Lustre: server umount lustre-OST03e8 complete [ 6877.881669] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 6887.605915] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 6898.479033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6904.961176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6911.010877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6916.425185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6922.738507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6922.802452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.050860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6923.079056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6923.127358] Lustre: lustre-MDT0000: new disk, initializing [ 6923.229545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.245669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6926.666720] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6935.741259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6935.812350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6935.883761] Lustre: 310500:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6935.947148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6935.951332] Lustre: Skipped 1 previous similar message [ 6936.003557] Lustre: lustre-MDT0001: new disk, initializing [ 6936.104525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6936.151372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6936.163628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6939.793687] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6945.754908] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6948.335864] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6953.918827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6953.984215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6954.156545] Lustre: lustre-OST0000: new disk, initializing [ 6954.169390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6954.250887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6959.623940] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 6962.746034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6962.755414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6962.890044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6967.131509] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6971.219694] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6971.417282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6975.245640] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6975.539958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6978.529796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6978.535644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6978.555100] LustreError: Skipped 1 previous similar message [ 6978.572454] Lustre: Skipped 1 previous similar message [ 6978.584604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6978.586506] Lustre: Skipped 2 previous similar messages [ 6983.617324] LustreError: 313438:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6983.633204] LustreError: 313438:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6983.663150] LustreError: 312714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6983.673813] LustreError: 312714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6983.817785] Lustre: server umount lustre-OST0000 complete [ 6988.769246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.773706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6988.815549] Lustre: Skipped 2 previous similar messages [ 6993.754481] LustreError: 313640:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6993.812394] LustreError: 313640:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6993.987738] LustreError: 310512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.013638] LustreError: 310512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6994.162453] Lustre: server umount lustre-MDT0000 complete [ 6999.010170] LustreError: 310512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.032261] LustreError: 310512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6999.339838] LustreError: 310493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764797905 with bad export cookie 11727150789346829992 [ 6999.344496] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6999.344715] LustreError: 310493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6999.712060] Lustre: server umount lustre-MDT0001 complete [ 7012.185569] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:38:35 (1764797915) [ 7014.050626] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7016.105214] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:38:39 (1764797919) [ 7020.181810] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7028.499725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7032.750310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7037.332476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7041.704759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7042.853179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7049.671599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.974506] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7049.989904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7050.214759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7054.967862] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7064.305854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7064.443722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7064.499893] Lustre: 315980:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7068.835019] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7074.466146] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7078.035089] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7085.343143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7085.433451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7085.724831] Lustre: Denying initial registration attempt from nid 192.168.201.105@tcp, specified as failover [ 7085.747634] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7085.763372] LustreError: 317597:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 7085.785646] LustreError: 317597:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 7085.796996] LustreError: 317597:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7085.999141] Lustre: server umount lustre-OST0000 complete [ 7086.001037] LustreError: 317597:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7095.782041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7095.792199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.811125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7095.817961] Lustre: Skipped 1 previous similar message [ 7098.798233] LustreError: 317966:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 7098.805703] LustreError: 317966:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7098.960439] Lustre: server umount lustre-MDT0000 complete [ 7101.423760] LustreError: 315991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7101.443064] LustreError: 315991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7102.261620] LustreError: 315972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798007 with bad export cookie 11727150789346830874 [ 7102.280963] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7102.636187] Lustre: server umount lustre-MDT0001 complete [ 7112.518683] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7114.834124] Key type lgssc unregistered [ 7115.086519] LNet: 318701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7115.094907] LNetError: 318701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7115.120970] LNet: Removed LNI 192.168.201.105@tcp [ 7115.947744] Key type .llcrypt unregistered [ 7115.949679] Key type ._llcrypt unregistered [ 7126.411607] Key type ._llcrypt registered [ 7126.420117] Key type .llcrypt registered [ 7126.549588] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7127.658148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7127.719990] alg: No test for adler32 (adler32-zlib) [ 7129.025542] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 7129.295753] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7131.009450] Key type lgssc registered [ 7132.528391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7140.662440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7145.358132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7149.732781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7154.126515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7155.325454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7170.497811] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7179.353749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7179.377087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.781203] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7180.807684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7180.985744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.650559] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7193.338990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7193.449936] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7193.479357] Lustre: 321464:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7193.717936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7197.602149] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7202.319663] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7205.167590] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7212.021691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7212.115118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7212.283005] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7212.591161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.743210] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7226.187324] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7232.484306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7232.492890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7232.504098] LustreError: Skipped 1 previous similar message [ 7232.504967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7232.509690] Lustre: Skipped 1 previous similar message [ 7232.533224] Lustre: Skipped 1 previous similar message [ 7234.505085] LustreError: 323747:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7234.729714] Lustre: server umount lustre-OST0000 complete [ 7239.651703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7239.665078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7239.679572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.725117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.731507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.760917] LustreError: 323950:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7244.763973] LustreError: 323950:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7244.902908] Lustre: server umount lustre-MDT0000 complete [ 7248.203145] LustreError: 321455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798153 with bad export cookie 11568180469826545752 [ 7248.209308] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7248.214425] LustreError: 321455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7248.351737] LustreError: 324152:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7248.361375] LustreError: 324152:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7248.564659] Lustre: server umount lustre-MDT0001 complete [ 7256.537585] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7258.938396] Key type lgssc unregistered [ 7259.219901] LNet: 324685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7259.232757] LNetError: 324685:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7260.263700] LNet: Removed LNI 192.168.201.105@tcp [ 7260.965534] Key type .llcrypt unregistered [ 7260.969792] Key type ._llcrypt unregistered [ 7282.272028] Key type ._llcrypt registered [ 7282.273714] Key type .llcrypt registered [ 7282.357108] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7283.429276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7283.438853] alg: No test for adler32 (adler32-zlib) [ 7284.518706] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 7284.854878] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7286.554400] Key type lgssc registered [ 7287.536449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7298.305156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7298.335348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.635254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.859826] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7310.820786] LustreError: 326078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.892971] LustreError: 326078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7315.940829] LustreError: 326078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.075992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7316.424992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7320.807423] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7327.409199] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7332.139243] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7339.736207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7340.176754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.617595] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7352.244831] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7362.018120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.023241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.048825] Lustre: Skipped 1 previous similar message [ 7362.050227] Lustre: Skipped 2 previous similar messages [ 7365.062547] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 7365.074718] Lustre: Skipped 1 previous similar message [ 7367.140162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.005527] LustreError: 328329:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7368.160257] Lustre: server umount lustre-MDT0000 complete [ 7370.126529] LustreError: 326074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.262287] LustreError: 326078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7372.337419] LustreError: 326078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7373.769319] LustreError: 326057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798279 with bad export cookie 5275596741845296663 [ 7373.795256] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7375.291996] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 7375.302643] Lustre: Skipped 2 previous similar messages [ 7377.377397] LustreError: 327191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7377.377706] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.401448] LustreError: 327191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7380.364099] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [ 7380.380466] Lustre: Skipped 2 previous similar messages [ 7385.486897] LustreError: 326074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.497353] LustreError: 326074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7388.128311] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7388.202206] LustreError: 328531:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7388.208719] LustreError: 328531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7388.341544] Lustre: server umount lustre-MDT0001 complete [ 7394.266485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7402.980396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7418.411911] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4936af490fe5c38a [ 7418.417646] Lustre: MGC192.168.201.105@tcp: Connection restored to 0@lo (at 0@lo) [ 7418.722326] LustreError: 327688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.821358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.905520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 7423.149568] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7432.128778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7432.655883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7437.038565] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7438.832424] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7442.685584] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7444.291754] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7448.018275] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7449.853450] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7458.277839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7458.281219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7458.307225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7458.310015] Lustre: Skipped 2 previous similar messages [ 7461.065897] LustreError: 330842:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7461.088952] LustreError: 330842:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7461.283944] Lustre: server umount lustre-OST0000 complete [ 7468.514296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7468.534424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7468.545068] Lustre: Skipped 1 previous similar message [ 7474.658546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7474.661064] Lustre: Skipped 5 previous similar messages [ 7479.268557] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7479.373100] LustreError: 331044:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7479.376416] LustreError: 331044:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7479.460345] Lustre: server umount lustre-MDT0000 complete [ 7479.801773] LustreError: 329341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7479.832741] LustreError: 329341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7484.496637] LustreError: 326059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798390 with bad export cookie 5275596741845304202 [ 7484.504715] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7484.523337] LustreError: 326059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7484.646411] LustreError: 331246:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7484.652500] LustreError: 331246:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7485.008805] Lustre: server umount lustre-MDT0001 complete [ 7496.554569] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7499.705070] Key type lgssc unregistered [ 7499.968355] LNet: 331779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7499.991838] LNetError: 331779:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7500.017687] LNet: Removed LNI 192.168.201.105@tcp [ 7500.965864] Key type .llcrypt unregistered [ 7500.968239] Key type ._llcrypt unregistered [ 7526.118142] Key type ._llcrypt registered [ 7526.119834] Key type .llcrypt registered [ 7526.261995] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7527.138824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7527.172348] alg: No test for adler32 (adler32-zlib) [ 7528.519573] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 7528.784563] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7530.488561] Key type lgssc registered [ 7531.834246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7540.541560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7540.557618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7542.049823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7546.070946] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7551.459876] LustreError: 333178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.497348] LustreError: 333178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7556.596941] LustreError: 333178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7557.318189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7557.954812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7563.007127] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7570.315319] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7574.215511] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7577.574048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7577.579724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7577.598629] Lustre: Skipped 1 previous similar message [ 7577.611144] Lustre: Skipped 1 previous similar message [ 7582.690302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7582.706940] Lustre: Skipped 1 previous similar message [ 7583.593491] LustreError: 334570:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7583.687576] Lustre: server umount lustre-MDT0000 complete [ 7583.717764] LustreError: 333172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7583.725697] LustreError: 333172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7586.584680] LustreError: 333158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798492 with bad export cookie 11528061691825973129 [ 7586.592517] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7586.745140] LustreError: 334774:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7586.748078] LustreError: 334774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7586.958325] Lustre: server umount lustre-MDT0001 complete [ 7602.335311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7602.669219] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7602.721917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7603.011592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.471092] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7616.184242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7616.282624] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7616.301613] Lustre: 335544:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7616.519289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7620.469503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7626.831736] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7632.834315] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7641.975336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7642.175035] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7642.190698] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7642.196772] LustreError: 337203:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 7642.201707] LustreError: 337203:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 7642.205289] LustreError: 337203:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7642.332943] Lustre: server umount lustre-OST0000 complete [ 7642.335085] LustreError: 337203:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7650.751764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7650.961850] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7651.205692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7657.029958] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7657.586991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 7663.328224] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7672.022814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7672.097501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7672.230572] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7672.364111] Lustre: lustre-OST0001: new disk, initializing [ 7672.389793] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7672.546523] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7678.075367] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7681.601444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7681.620028] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7681.739960] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7683.912578] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7687.938121] LustreError: 339328:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7687.969843] LustreError: 339328:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7688.122065] Lustre: server umount lustre-OST0001 complete [ 7692.284886] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7692.288424] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.320806] LustreError: 338643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.326333] Lustre: Skipped 1 previous similar message [ 7692.386955] LustreError: 338643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7693.798654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7693.820814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.850147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7697.397083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7697.397441] LustreError: 337673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.400506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.418663] LustreError: 337673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7699.800723] LustreError: 339529:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7699.828019] LustreError: 339529:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7700.112932] Lustre: server umount lustre-OST0000 complete [ 7707.620431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7707.626652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7707.639803] Lustre: Skipped 2 previous similar messages [ 7710.753714] LustreError: 339731:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7710.780562] LustreError: 339731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7711.091515] Lustre: server umount lustre-MDT0000 complete [ 7712.737891] LustreError: 339298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7712.757504] LustreError: 339298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7715.081299] LustreError: 335535:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798620 with bad export cookie 11528061691825973682 [ 7715.084544] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7715.088940] LustreError: 335535:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7715.535465] Lustre: server umount lustre-MDT0001 complete [ 7726.020741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7728.391355] Key type lgssc unregistered [ 7728.681515] LNet: 340465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7728.690586] LNetError: 340465:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7729.768518] LNet: Removed LNI 192.168.201.105@tcp [ 7730.633520] Key type .llcrypt unregistered [ 7730.636060] Key type ._llcrypt unregistered [ 7761.941853] Key type ._llcrypt registered [ 7761.945886] Key type .llcrypt registered [ 7762.063586] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7775.744770] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7777.310511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7777.368230] alg: No test for adler32 (adler32-zlib) [ 7778.529380] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 7778.803428] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7780.480433] Key type lgssc registered [ 7781.921083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7791.012217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7797.220734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7803.020943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7809.616212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7815.934571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7815.979815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7815.989419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.206337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7817.249826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7817.327338] Lustre: lustre-MDT0000: new disk, initializing [ 7817.404300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7817.452365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7820.597149] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7830.505317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7830.577949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7830.674828] Lustre: 344418:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7830.725564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7830.738305] Lustre: Skipped 1 previous similar message [ 7830.833863] Lustre: lustre-MDT0001: new disk, initializing [ 7830.933077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7830.980374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7830.997667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7834.408360] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7840.986876] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7844.198440] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7850.010466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7850.066312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7850.228807] Lustre: lustre-OST0000: new disk, initializing [ 7850.231868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7850.313726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.333741] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7858.778447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7858.798836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7858.965651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7863.974158] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7869.068809] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7869.314474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7874.920704] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7875.170339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7879.652357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7879.655535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7879.658618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7879.678432] Lustre: Skipped 1 previous similar message [ 7883.528130] LustreError: 347358:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7883.730415] Lustre: server umount lustre-OST0000 complete [ 7889.892324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7889.900124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7889.905876] Lustre: Skipped 2 previous similar messages [ 7893.669231] LustreError: 347560:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7893.676919] LustreError: 347560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7893.793724] Lustre: server umount lustre-MDT0000 complete [ 7896.672649] LustreError: 344409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798802 with bad export cookie 17752948567829567209 [ 7896.675269] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7896.689917] LustreError: 344409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7896.800067] LustreError: 347761:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7896.807148] LustreError: 347761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7896.958642] Lustre: server umount lustre-MDT0001 complete [ 7904.906430] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 16:53:29 (1764798809) [ 7911.645027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7917.716212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7941.118502] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 7951.042735] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7962.211271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7971.967369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7980.270203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7987.752398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7994.560956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7994.640514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7995.006283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7995.067170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7995.180222] Lustre: lustre-MDT0000: new disk, initializing [ 7995.314877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7995.341514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7999.676677] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8010.785458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8010.865642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8010.941226] Lustre: 351538:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8011.000168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8011.018363] Lustre: Skipped 1 previous similar message [ 8011.187582] Lustre: lustre-MDT0001: new disk, initializing [ 8011.310396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8011.345038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8011.358231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8015.778211] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8024.287768] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8029.038670] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8037.443245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8037.596608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8037.855947] Lustre: lustre-OST0000: new disk, initializing [ 8037.866962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8037.941391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8045.787052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8045.807867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8046.071474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8046.528350] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8058.038545] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8063.695505] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8063.958737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8067.959191] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8068.233864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8071.651242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8071.657760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8071.675251] Lustre: Skipped 1 previous similar message [ 8071.676727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8071.680328] LustreError: Skipped 1 previous similar message [ 8071.687530] Lustre: Skipped 1 previous similar message [ 8076.106112] LustreError: 354481:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8076.114932] LustreError: 354481:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8076.309901] Lustre: server umount lustre-OST0000 complete [ 8081.889552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.895686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8081.907820] Lustre: Skipped 2 previous similar messages [ 8086.373143] LustreError: 354684:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8086.387758] LustreError: 354684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8086.584995] Lustre: server umount lustre-MDT0000 complete [ 8087.010042] LustreError: 351547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8087.015630] LustreError: 351547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8090.290344] LustreError: 352320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764798996 with bad export cookie 17752948567829568098 [ 8090.302153] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8090.322835] LustreError: 352320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8090.650627] Lustre: server umount lustre-MDT0001 complete [ 8098.822411] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:56:42 (1764799002) [ 8105.322221] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:56:49 (1764799009) [ 8111.605884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.954747] LustreError: 355902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.967583] LustreError: 355902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8112.046627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8115.093753] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8117.232050] LustreError: 355903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.250907] LustreError: 355903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8121.315027] LustreError: 355903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8123.074356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8127.172443] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8132.636934] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8136.377086] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8143.430271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8149.220656] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8154.619294] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8166.562629] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8167.969144] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8169.515637] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8170.858689] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8172.527102] Lustre: Failing over lustre-MDT0000 [ 8172.690672] LustreError: 358364:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8172.696655] LustreError: 358364:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8172.732439] LustreError: 358364:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (000000003f8578c5) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8172.819038] Lustre: server umount lustre-MDT0000 complete [ 8174.563572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8174.572473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.598200] LustreError: 356924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.753681] LustreError: 356924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.775561] LustreError: 356924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 8190.431823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8190.506781] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8190.711530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8190.719705] Lustre: Skipped 2 previous similar messages [ 8190.758748] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8191.882153] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8194.296461] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8196.077518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8196.114542] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8196.154287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8200.359771] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8201.758536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8205.129813] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8207.076583] Lustre: Failing over lustre-MDT0000 [ 8207.198219] LustreError: 359443:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8207.211023] LustreError: 359443:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8207.263343] LustreError: 355897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.286091] LustreError: 355897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8207.395584] Lustre: server umount lustre-MDT0000 complete [ 8211.429945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.436226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8211.436361] Lustre: Skipped 2 previous similar messages [ 8225.832953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.998300] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8226.270777] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8227.725296] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8230.358971] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8231.404915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8231.412729] Lustre: Skipped 2 previous similar messages [ 8231.430399] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8231.502773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8237.314265] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8238.986513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8246.765988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8246.776362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.778273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.778280] Lustre: Skipped 1 previous similar message [ 8246.805415] Lustre: Skipped 4 previous similar messages [ 8249.769051] LustreError: 360473:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8249.777085] LustreError: 360473:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8249.889195] Lustre: server umount lustre-MDT0000 complete [ 8251.882943] LustreError: 356924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.915986] LustreError: 356924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 8253.392528] LustreError: 356621:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799159 with bad export cookie 17752948567829570793 [ 8253.398030] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8253.409096] LustreError: 356621:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8253.701138] Lustre: server umount lustre-MDT0001 complete [ 8268.097886] Lustre: server umount lustre-OST0000 complete [ 8289.145666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.775857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8289.788197] Lustre: Skipped 1 previous similar message [ 8295.287708] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8307.134348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8311.310166] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8316.327257] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8320.602285] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8330.867281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8335.482521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8340.011884] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8347.795470] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8353.254443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8353.260427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.270956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8353.274731] Lustre: Skipped 2 previous similar messages [ 8354.938186] LustreError: 363951:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8354.941394] LustreError: 363951:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8355.042253] Lustre: server umount lustre-OST0000 complete [ 8371.686980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8371.700162] Lustre: Skipped 6 previous similar messages [ 8373.216127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8373.400912] Lustre: server umount lustre-MDT0000 complete [ 8376.803173] LustreError: 361707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.819042] LustreError: 361707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8376.899531] LustreError: 361688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799282 with bad export cookie 17752948567829571500 [ 8376.909457] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8376.917491] LustreError: 361688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8377.257196] Lustre: server umount lustre-MDT0001 complete [ 8386.477463] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8389.415251] Key type lgssc unregistered [ 8389.697224] LNet: 364903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8389.708221] LNetError: 364903:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8389.721967] LNet: Removed LNI 192.168.201.105@tcp [ 8390.620253] Key type .llcrypt unregistered [ 8390.625158] Key type ._llcrypt unregistered [ 8418.685992] Key type ._llcrypt registered [ 8418.691258] Key type .llcrypt registered [ 8418.798674] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8419.953452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8419.978927] alg: No test for adler32 (adler32-zlib) [ 8421.152898] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 8421.476232] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8423.152178] Key type lgssc registered [ 8424.385163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8432.969637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8433.000586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.619546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.346612] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8444.919037] LustreError: 366401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.946130] LustreError: 366401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8448.329314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8448.971213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8453.670353] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8460.294896] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8464.685987] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8471.380150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8471.753729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8477.404934] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8479.981347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8482.853348] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8495.074666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8495.092677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.110274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8495.596544] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.611151] Lustre: Skipped 1 previous similar message [ 8495.616217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8495.619295] Lustre: Skipped 1 previous similar message [ 8497.851941] LustreError: 368650:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8497.974799] Lustre: server umount lustre-MDT0000 complete [ 8500.707706] LustreError: 366400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.733182] LustreError: 366400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8501.902646] LustreError: 366382:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799407 with bad export cookie 1478105745135819724 [ 8501.909139] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8502.040283] LustreError: 368852:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8502.046728] LustreError: 368852:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8502.325409] Lustre: server umount lustre-MDT0001 complete [ 8515.525317] LustreError: 369053:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8515.535917] LustreError: 369053:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8515.728539] Lustre: server umount lustre-OST0000 complete [ 8529.528422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8529.937931] LustreError: 369784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.034813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8534.271642] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8535.530845] LustreError: 369783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8535.552191] LustreError: 369783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8539.620397] LustreError: 369783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8544.388105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8544.795932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8548.616271] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8553.889246] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8558.233656] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8564.786876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8565.064892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8570.682605] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8574.430825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8577.403038] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8594.915439] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.917685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.007164] Lustre: Skipped 2 previous similar messages [ 8595.029358] Lustre: Skipped 2 previous similar messages [ 8597.348869] LustreError: 372135:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8597.360834] LustreError: 372135:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8597.471870] Lustre: server umount lustre-MDT0000 complete [ 8600.048727] LustreError: 369783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.082014] LustreError: 369783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8602.050140] LustreError: 369765:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799507 with bad export cookie 1478105745135820830 [ 8602.060140] LustreError: 369765:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8602.068406] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8602.202762] LustreError: 372336:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8602.215228] LustreError: 372336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8602.637098] Lustre: server umount lustre-MDT0001 complete [ 8617.165324] LustreError: 372538:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8617.173721] LustreError: 372538:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8617.357646] Lustre: server umount lustre-OST0000 complete [ 8631.314673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.635107] LustreError: 373366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8631.692186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8634.856364] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8642.397113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8646.642752] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8652.761483] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8656.888972] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8663.801615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8664.238360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8664.245977] Lustre: Skipped 1 previous similar message [ 8669.881839] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8674.877799] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8676.540841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8689.123077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8689.138924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.166162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8692.208330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8694.142114] LustreError: 375629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8694.148384] LustreError: 375629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8694.336218] Lustre: server umount lustre-OST0000 complete [ 8699.364459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8699.374363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.381370] Lustre: Skipped 1 previous similar message [ 8699.401280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8699.403676] Lustre: Skipped 2 previous similar messages [ 8704.393335] Lustre: server umount lustre-MDT0000 complete [ 8707.147737] LustreError: 373347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799612 with bad export cookie 1478105745135821859 [ 8707.154988] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8707.166815] LustreError: 373347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8707.471955] Lustre: server umount lustre-MDT0001 complete [ 8717.590616] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8720.274228] Key type lgssc unregistered [ 8720.575642] LNet: 376566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8720.590631] LNetError: 376566:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8720.609063] LNet: Removed LNI 192.168.201.105@tcp [ 8721.324495] Key type .llcrypt unregistered [ 8721.329510] Key type ._llcrypt unregistered [ 8746.653167] Key type ._llcrypt registered [ 8746.657183] Key type .llcrypt registered [ 8746.749333] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8747.858212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8747.868084] alg: No test for adler32 (adler32-zlib) [ 8748.924787] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 8749.162195] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8750.848218] Key type lgssc registered [ 8752.310803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8760.008130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8760.040078] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8760.050094] LustreError: 378012:0:(osd_handler.c:8573:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8760.067748] LustreError: 378012:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8760.083232] LustreError: 378012:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8760.088058] LustreError: 378012:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8760.094563] LustreError: 378012:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8766.697460] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8766.702643] LustreError: 378410:0:(osd_handler.c:8573:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8766.721694] LustreError: 378410:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8766.724526] LustreError: 378410:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8766.736641] LustreError: 378410:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8766.748243] LustreError: 378410:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8787.100507] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8789.988919] Key type lgssc unregistered [ 8790.314450] LNet: 379396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8790.324048] LNetError: 379396:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8790.334643] LNet: Removed LNI 192.168.201.105@tcp [ 8791.064753] Key type .llcrypt unregistered [ 8791.066507] Key type ._llcrypt unregistered [ 8807.297869] Key type ._llcrypt registered [ 8807.300103] Key type .llcrypt registered [ 8807.430511] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8822.103992] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8822.880491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8823.018969] alg: No test for adler32 (adler32-zlib) [ 8824.025173] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 8824.223349] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8825.848204] Key type lgssc registered [ 8826.706233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8833.632158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8838.633102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8842.490852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8846.497247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8851.693858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8851.741713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8851.757594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8853.007404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8853.043526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8853.125246] Lustre: lustre-MDT0000: new disk, initializing [ 8853.233229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8853.260414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8857.055586] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8867.545348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8867.639433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8867.753076] Lustre: 382530:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8867.813873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8867.816378] Lustre: Skipped 1 previous similar message [ 8867.863510] Lustre: lustre-MDT0001: new disk, initializing [ 8867.988324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8868.021307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8868.039885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8873.096057] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8880.168201] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8884.606508] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8891.601994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8891.731733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8892.009241] Lustre: lustre-OST0000: new disk, initializing [ 8892.023768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8892.135172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8897.664768] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8899.138896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8899.149987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8899.224317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8904.934791] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8908.562665] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8908.780805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8912.509295] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8912.741732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8914.916154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8914.921220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8914.923473] LustreError: Skipped 1 previous similar message [ 8914.925110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8914.938827] Lustre: Skipped 1 previous similar message [ 8920.033805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8920.047107] Lustre: Skipped 2 previous similar messages [ 8920.704265] LustreError: 385469:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8920.898896] Lustre: server umount lustre-OST0000 complete [ 8925.153458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8925.162261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.274040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.277130] Lustre: Skipped 2 previous similar messages [ 8930.818248] LustreError: 385670:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8930.829614] LustreError: 385670:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8930.954217] Lustre: server umount lustre-MDT0000 complete [ 8934.930968] LustreError: 382522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799840 with bad export cookie 9439778841185553402 [ 8934.943707] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8934.959476] LustreError: 382522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8935.189295] LustreError: 385872:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8935.201350] LustreError: 385872:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8935.534431] Lustre: server umount lustre-MDT0001 complete [ 8945.029066] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8946.836980] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:10:50 (1764799850) [ 8953.725521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.383124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8957.984149] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8959.460613] LustreError: 386592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.473632] LustreError: 386592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8964.579948] LustreError: 386592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8965.688418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8966.027758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8970.355126] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8975.905626] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8979.608423] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8986.778665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8987.217143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8993.826324] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8999.910488] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9007.102724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9007.167834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9007.267213] Lustre: lustre-OST0001: new disk, initializing [ 9007.276591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9007.363887] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9013.563733] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9016.922024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9016.934681] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9017.080617] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9021.841942] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9027.564469] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.588214] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9027.589878] Lustre: Skipped 2 previous similar messages [ 9032.317401] LustreError: 389793:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9032.325425] LustreError: 389793:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9032.467554] Lustre: server umount lustre-OST0001 complete [ 9032.678513] LustreError: 388191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.686231] LustreError: 388191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9037.801954] LustreError: 388189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.813270] LustreError: 388189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9042.920969] LustreError: 388189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.955033] LustreError: 388189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9051.109341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9051.123075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.145198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.147897] Lustre: Skipped 2 previous similar messages [ 9052.129330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9053.159379] LustreError: 388190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9053.196353] LustreError: 388190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9062.880138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9062.962109] LustreError: 390019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9062.965129] LustreError: 390019:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9063.356722] Lustre: server umount lustre-OST0000 complete [ 9068.517600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.518598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9068.518826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9068.525977] Lustre: Skipped 2 previous similar messages [ 9068.528804] Lustre: Skipped 5 previous similar messages [ 9082.336235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9082.472822] LustreError: 390222:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9082.490775] LustreError: 390222:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9082.624450] Lustre: server umount lustre-MDT0000 complete [ 9083.877984] LustreError: 388408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9083.896367] LustreError: 388408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9087.137993] LustreError: 386573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764799992 with bad export cookie 9439778841185554284 [ 9087.142590] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9087.165838] LustreError: 386573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9093.750531] Lustre: server umount lustre-MDT0001 complete [ 9104.843733] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9107.545913] Key type lgssc unregistered [ 9107.861839] LNet: 390957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9107.871191] LNetError: 390957:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9108.912985] LNet: Removed LNI 192.168.201.105@tcp [ 9109.756227] Key type .llcrypt unregistered [ 9109.758885] Key type ._llcrypt unregistered [ 9145.523054] Key type ._llcrypt registered [ 9145.524550] Key type .llcrypt registered [ 9145.634060] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9159.386639] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9160.451547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9160.596937] alg: No test for adler32 (adler32-zlib) [ 9161.682137] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 9161.903432] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9163.552212] Key type lgssc registered [ 9164.662315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9173.488535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9179.814744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9185.827948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9191.833845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9198.418530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9198.463984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9198.485431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.728235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9199.773193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9199.818716] Lustre: lustre-MDT0000: new disk, initializing [ 9199.909918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9199.935414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9203.952456] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9213.875791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9213.991609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9214.143313] Lustre: 394909:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9214.197905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9214.200715] Lustre: Skipped 1 previous similar message [ 9214.253915] Lustre: lustre-MDT0001: new disk, initializing [ 9214.392635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9214.429155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9214.442405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9218.416925] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9226.105980] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9230.321836] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9236.976853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9237.060795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9237.375723] Lustre: lustre-OST0000: new disk, initializing [ 9237.380486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9237.483254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9242.939179] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9247.334106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9247.348542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9247.465895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9250.400382] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9254.785931] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9255.005127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9259.168535] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9259.399688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9263.079897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9263.085475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9263.091470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9263.130752] Lustre: Skipped 1 previous similar message [ 9267.647603] LustreError: 397850:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9267.875524] Lustre: server umount lustre-OST0000 complete [ 9273.314179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9273.340451] Lustre: Skipped 1 previous similar message [ 9273.353028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9273.355854] Lustre: Skipped 1 previous similar message [ 9277.538244] LustreError: 398052:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9277.544847] LustreError: 398052:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9277.714710] Lustre: server umount lustre-MDT0000 complete [ 9281.803587] LustreError: 394899:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764800187 with bad export cookie 5401680908321279490 [ 9281.808956] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9281.826604] LustreError: 394899:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9282.120372] LustreError: 398253:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9282.128887] LustreError: 398253:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9282.475225] Lustre: server umount lustre-MDT0001 complete [ 9291.572413] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 17:16:35 (1764800195) [ 9301.668829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9311.242971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.649147] Lustre: lustre-MDT0000: new disk, initializing [ 9311.717319] LustreError: 399522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9311.759746] LustreError: 399522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9311.814889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9316.035402] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9316.838266] LustreError: 399522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9321.955530] LustreError: 399522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9325.438330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9325.733933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9330.388495] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9336.559578] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9340.980981] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9346.018100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9346.021424] LustreError: Skipped 1 previous similar message [ 9346.043453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9346.075201] Lustre: Skipped 1 previous similar message [ 9346.089965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9349.680322] LustreError: 400865:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9349.713273] LustreError: 400865:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9350.042752] Lustre: server umount lustre-MDT0000 complete [ 9350.649554] LustreError: 399521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9350.697524] LustreError: 399521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9354.228833] LustreError: 400866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764800259 with bad export cookie 5401680908321280372 [ 9354.232828] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9354.250137] LustreError: 400866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9354.466758] LustreError: 401067:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9354.469889] LustreError: 401067:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9354.867338] Lustre: server umount lustre-MDT0001 complete [ 9366.305586] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:17:49 (1764800269) [ 9372.516646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.963191] LustreError: 401788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.036739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9376.474445] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9382.372900] LustreError: 401788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9382.398620] LustreError: 401788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9385.825399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9390.132273] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9397.331420] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9400.893072] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9408.128241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9408.687722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.690940] Lustre: Skipped 1 previous similar message [ 9414.443786] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9420.187231] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9427.844336] Lustre: Permanently deactivating lustre-OST0000 [ 9427.863800] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9433.679339] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9435.692440] LustreError: 404384:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9435.696660] LustreError: 404384:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 9435.705285] LustreError: 404384:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9438.590041] LustreError: 404432:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 9438.593641] LustreError: 404432:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9442.886146] LustreError: 404529:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9442.902830] LustreError: 404529:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9443.201894] Lustre: server umount lustre-OST0000 complete [ 9445.885401] LustreError: 404682:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 9445.889751] LustreError: 404682:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9447.911709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9447.920581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9447.940227] Lustre: Skipped 1 previous similar message [ 9447.944845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9447.947800] Lustre: Skipped 1 previous similar message [ 9454.050093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9454.052814] Lustre: Skipped 2 previous similar messages [ 9462.240267] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9462.331434] LustreError: 404779:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9462.345407] LustreError: 404779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9462.438976] Lustre: server umount lustre-MDT0000 complete [ 9464.291112] LustreError: 402501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9464.312681] LustreError: 402501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9465.957900] LustreError: 404136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764800371 with bad export cookie 5401680908321280925 [ 9465.963194] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9465.990172] LustreError: 404136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9475.738261] Lustre: server umount lustre-MDT0001 complete [ 9484.432225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9488.733796] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9491.435143] LustreError: 405826:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9497.354510] Lustre: 406165:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.105@tcp [ 9499.327702] LustreError: 406262:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9499.331209] LustreError: 406262:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9499.399158] Lustre: server umount MGS complete [ 9510.860259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.198188] LustreError: 406890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9511.215340] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9511.219117] Lustre: Skipped 1 previous similar message [ 9511.276152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9514.893274] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9522.648513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9522.881959] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9525.854492] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9529.794150] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9533.186582] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9534.810994] Lustre: Permanently reactivating lustre-OST0000 [ 9534.824953] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9534.828706] Lustre: Skipped 2 previous similar messages [ 9564.129876] LustreError: 408500:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9569.514878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9574.360421] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9579.041450] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9584.099018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9584.102727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9584.108886] LustreError: Skipped 1 previous similar message [ 9584.110183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9584.144534] Lustre: Skipped 2 previous similar messages [ 9584.160049] Lustre: Skipped 3 previous similar messages [ 9588.362088] LustreError: 410104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9588.368584] LustreError: 410104:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9588.568391] Lustre: server umount lustre-OST0000 complete [ 9594.338662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9594.349420] Lustre: Skipped 1 previous similar message [ 9598.297903] Lustre: server umount lustre-MDT0000 complete [ 9599.457000] LustreError: 406890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9599.471724] LustreError: 406890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9601.391106] LustreError: 406872:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764800507 with bad export cookie 5401680908321281884 [ 9601.397044] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9601.404457] LustreError: 406872:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9601.723862] Lustre: server umount lustre-MDT0001 complete [ 9609.997614] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9611.654618] Key type lgssc unregistered [ 9611.825909] LNet: 411045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9611.831514] LNetError: 411045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9611.840717] LNet: Removed LNI 192.168.201.105@tcp [ 9612.314699] Key type .llcrypt unregistered [ 9612.317124] Key type ._llcrypt unregistered [ 9628.605580] Key type ._llcrypt registered [ 9628.611684] Key type .llcrypt registered [ 9628.754102] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9643.849977] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9645.178329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9645.220094] alg: No test for adler32 (adler32-zlib) [ 9646.200663] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 9646.370409] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9648.056466] Key type lgssc registered [ 9649.166941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9658.546447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9666.233219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9672.671993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9679.074836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9686.360712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9686.425588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9686.444087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.732319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9687.796530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9687.977247] Lustre: lustre-MDT0000: new disk, initializing [ 9688.137676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.180754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9692.622875] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9702.413983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9702.477859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9702.562443] Lustre: 414178:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9702.634341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9702.637188] Lustre: Skipped 1 previous similar message [ 9702.699420] Lustre: lustre-MDT0001: new disk, initializing [ 9702.802089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9702.852701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9702.867269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9706.413349] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9713.399189] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9716.411379] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9722.197027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9722.259961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9722.513333] Lustre: lustre-OST0000: new disk, initializing [ 9722.520114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9722.647528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9727.945387] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9731.638958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9731.649323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9731.722944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9733.834843] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9737.380378] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9737.554994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9741.138694] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9741.306751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9747.425698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9747.435399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.450529] LustreError: Skipped 1 previous similar message [ 9747.451614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9747.481897] Lustre: Skipped 1 previous similar message [ 9752.558271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9752.561044] Lustre: Skipped 1 previous similar message [ 9757.152158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9757.186059] LustreError: 417119:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9757.364150] Lustre: server umount lustre-OST0000 complete [ 9762.792278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.801956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9767.558751] LustreError: 417323:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9767.580030] LustreError: 417323:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9767.744661] Lustre: server umount lustre-MDT0000 complete [ 9772.194980] LustreError: 414960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764800677 with bad export cookie 11683781258446720757 [ 9772.197958] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.255722] LustreError: 414960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9772.454701] LustreError: 417524:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9772.465635] LustreError: 417524:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9772.771602] Lustre: server umount lustre-MDT0001 complete [ 9781.865451] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9783.524978] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:24:47 (1764800687) [ 9789.991223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9790.402135] LustreError: 418244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9790.422387] LustreError: 418244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9790.504445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9794.177950] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9795.555880] LustreError: 418244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9800.680140] LustreError: 418243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9801.509585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9801.895521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9805.265116] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9810.257383] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9813.991959] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9819.927854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9820.233413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9825.180359] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9829.136802] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9830.821534] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9837.845096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9845.733306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.735570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9845.766045] Lustre: Skipped 1 previous similar message [ 9845.770905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9845.770914] Lustre: Skipped 1 previous similar message [ 9848.780316] LustreError: 420684:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9848.800553] LustreError: 420684:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9849.077183] Lustre: server umount lustre-OST0000 complete [ 9853.409535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9853.422515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9853.440321] Lustre: Skipped 1 previous similar message [ 9853.447869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9853.453917] Lustre: Skipped 1 previous similar message [ 9866.210350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9866.215036] Lustre: Skipped 5 previous similar messages [ 9867.232365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9867.294116] LustreError: 420887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9867.301919] LustreError: 420887:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9867.344028] Lustre: server umount lustre-MDT0000 complete [ 9871.045225] LustreError: 418225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764800776 with bad export cookie 11683781258446721632 [ 9871.052758] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9871.059421] LustreError: 418225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9871.512509] Lustre: server umount lustre-MDT0001 complete [ 9881.110357] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9883.757771] Key type lgssc unregistered [ 9884.023755] LNet: 421623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9884.035130] LNetError: 421623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9884.055976] LNet: Removed LNI 192.168.201.105@tcp [ 9884.991085] Key type .llcrypt unregistered [ 9885.000706] Key type ._llcrypt unregistered [ 9907.647699] Key type ._llcrypt registered [ 9907.649017] Key type .llcrypt registered [ 9907.746636] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9908.011821] Key type .llcrypt unregistered [ 9908.014473] Key type ._llcrypt unregistered [ 9921.864585] Key type ._llcrypt registered [ 9921.866125] Key type .llcrypt registered [ 9921.970632] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9922.753879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9922.889788] alg: No test for adler32 (adler32-zlib) [ 9923.993097] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [ 9924.186900] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9925.880424] Key type lgssc registered [ 9926.962728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9933.812292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9933.833969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9935.229904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9938.415840] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9944.547646] LustreError: 423774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.566237] LustreError: 423774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9945.690720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9945.936934] LustreError: 423774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9945.951650] LustreError: 423774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9945.955813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.953397] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9951.200928] LustreError: 423774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9951.227866] LustreError: 423774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9955.425889] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9956.321469] LustreError: 423770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.333154] LustreError: 423770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9959.405977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9961.429521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9961.809156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9965.272669] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9968.722643] Lustre: 423770:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 9968.734475] Lustre: 423770:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9968.739271] Lustre: 423770:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 9968.742923] Lustre: 423770:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 9968.761513] Lustre: 423770:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9968.766244] Lustre: 423770:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9974.349126] LustreError: 425828:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 9974.530707] Lustre: server umount lustre-OST0000 complete [ 9980.903318] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9980.906653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9980.918497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.433401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.435146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9987.559317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9987.562024] Lustre: Skipped 1 previous similar message [ 9987.579568] LustreError: 426029:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9987.579577] LustreError: 426029:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9987.673798] Lustre: server umount lustre-MDT0000 complete [ 9991.110036] LustreError: 423754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764800896 with bad export cookie 10899750623924648379 [ 9991.114928] LustreError: 423754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9991.118665] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10000.363992] LustreError: 426232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10000.370371] LustreError: 426232:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10000.584333] Lustre: server umount lustre-MDT0001 complete [10010.066450] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10012.808945] Key type lgssc unregistered [10013.226174] LNet: 426765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10013.234104] LNetError: 426765:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10013.243143] LNet: Removed LNI 192.168.201.105@tcp [10014.305621] Key type .llcrypt unregistered [10014.307703] Key type ._llcrypt unregistered [10039.842743] Key type ._llcrypt registered [10039.844160] Key type .llcrypt registered [10039.960423] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10040.970654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10041.005511] alg: No test for adler32 (adler32-zlib) [10042.178428] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [10042.390073] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10044.080197] Key type lgssc registered [10045.303416] Lustre: Echo OBD driver; http://www.lustre.org/ [10053.757219] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10053.772630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10069.282847] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10074.401496] LustreError: 428136:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [10101.152253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10107.380772] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10112.981613] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10119.627339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10120.209228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10120.290271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10123.976224] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10125.284461] LustreError: 428162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10125.292420] LustreError: 428162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10130.417463] LustreError: 428162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10133.212782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10133.583712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10137.733114] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10143.351714] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10147.778931] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10150.771515] Lustre: 429060:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [10150.781329] Lustre: 429060:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10150.789702] Lustre: 429060:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [10150.793810] Lustre: 429060:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [10150.802855] Lustre: 429060:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10150.809203] Lustre: 429060:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10156.004865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10156.017584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.024134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10156.383681] LustreError: 430416:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [10156.533118] Lustre: server umount lustre-OST0000 complete [10161.635643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10161.642360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10161.692256] Lustre: Skipped 1 previous similar message [10166.754307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10166.767283] Lustre: Skipped 2 previous similar messages [10167.054167] LustreError: 430617:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10167.123578] LustreError: 430617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10167.529190] Lustre: server umount lustre-MDT0000 complete [10171.499233] LustreError: 428143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764801077 with bad export cookie 5181815624754424074 [10171.536837] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10182.137282] LustreError: 430819:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10182.150280] LustreError: 430819:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10182.442265] Lustre: server umount lustre-MDT0001 complete [10194.140053] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10196.703930] Key type lgssc unregistered [10197.019591] LNet: 431355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10197.026100] LNetError: 431355:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10197.042154] LNet: Removed LNI 192.168.201.105@tcp [10197.984618] Key type .llcrypt unregistered [10197.987280] Key type ._llcrypt unregistered [10224.328513] Key type ._llcrypt registered [10224.335122] Key type .llcrypt registered [10224.473938] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10225.608706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10225.665123] alg: No test for adler32 (adler32-zlib) [10226.698934] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [10226.849542] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10228.496407] Key type lgssc registered [10229.464941] Lustre: Echo OBD driver; http://www.lustre.org/ [10236.991516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10237.027272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10238.586403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10242.254283] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10249.200415] LustreError: 432754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10249.219195] LustreError: 432754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10251.150986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10251.725841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10256.315572] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10262.332703] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10266.766697] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10274.356880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10274.698593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10280.025530] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10283.970251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10286.885489] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10292.706419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10292.715743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.727326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10294.243969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10294.249067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10294.253605] LustreError: 435000:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10294.272080] Lustre: Skipped 1 previous similar message [10294.275766] Lustre: Skipped 1 previous similar message [10294.311066] LustreError: 432747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10294.429953] Lustre: server umount lustre-MDT0000 complete [10297.195869] LustreError: 432734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.5@tcp arrived at 1764801202 with bad export cookie 10103146837228795350 [10299.364754] LustreError: 432753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10299.398473] LustreError: 432753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10299.520781] LustreError: 432734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764801205 with bad export cookie 10103146837228794454 [10299.536223] LustreError: 432734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10299.542630] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10302.436160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10302.449384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10302.465069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10304.484478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10304.484711] LustreError: 432750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.510446] LustreError: 432750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10305.689521] LustreError: 435203:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10305.712503] LustreError: 435203:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10306.061863] Lustre: server umount lustre-OST0000 complete [10309.601966] LustreError: 432750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.627843] LustreError: 432750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10313.741424] LustreError: 435558:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10313.748178] LustreError: 435558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10313.967819] Lustre: server umount lustre-MDT0001 complete [10325.400756] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10328.098319] Key type lgssc unregistered [10328.406670] LNet: 436090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10328.411711] LNetError: 436090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10329.450822] LNet: Removed LNI 192.168.201.105@tcp [10330.212347] Key type .llcrypt unregistered [10330.215640] Key type ._llcrypt unregistered [10355.510605] Key type ._llcrypt registered [10355.512394] Key type .llcrypt registered [10355.623645] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10356.686123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10356.719839] alg: No test for adler32 (adler32-zlib) [10357.740883] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [10357.976108] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10359.632179] Key type lgssc registered [10360.667773] Lustre: Echo OBD driver; http://www.lustre.org/ [10367.866269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10367.881706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10369.307076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10372.607584] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10379.674708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10379.747972] LustreError: 437488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.755776] LustreError: 437488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10380.130536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10383.984091] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10390.156116] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10394.603197] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10401.463900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10401.783459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10406.708903] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10411.340234] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10415.099371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10420.198683] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10420.213976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10423.228065] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [10423.241229] Lustre: Skipped 1 previous similar message [10425.316956] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10425.327260] Lustre: Skipped 1 previous similar message [10426.294832] LustreError: 439734:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10426.373317] Lustre: server umount lustre-MDT0001 complete [10428.304645] LustreError: 437485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.440290] LustreError: 437483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.468651] LustreError: 437483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10435.555031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10435.555575] LustreError: 437489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.566225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10435.566234] Lustre: Skipped 1 previous similar message [10435.567115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10437.565915] LustreError: 439937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10437.579544] LustreError: 439937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10437.733946] Lustre: server umount lustre-OST0000 complete [10440.675365] LustreError: 437489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10441.337877] LustreError: 440139:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10441.346848] LustreError: 440139:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10441.592674] Lustre: server umount lustre-MDT0000 complete [10453.127754] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10456.089792] Key type lgssc unregistered [10456.329498] LNet: 440825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10456.343029] LNetError: 440825:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10456.356163] LNet: Removed LNI 192.168.201.105@tcp [10457.130640] Key type .llcrypt unregistered [10457.132475] Key type ._llcrypt unregistered [10480.805079] Key type ._llcrypt registered [10480.806654] Key type .llcrypt registered [10480.891534] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [10494.048432] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10495.315967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10495.338221] alg: No test for adler32 (adler32-zlib) [10496.454898] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [10496.665114] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10498.361083] Key type lgssc registered [10499.455524] Lustre: Echo OBD driver; http://www.lustre.org/ [10508.662542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10515.678791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10521.627298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10527.954132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10537.975476] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10547.023268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10553.504418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10553.562821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10553.587087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10554.954734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10555.008233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10555.079912] Lustre: lustre-MDT0000: new disk, initializing [10555.178520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.214683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10559.198251] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10567.205914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10574.194086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10574.269811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10574.645539] Lustre: lustre-OST0000: new disk, initializing [10574.657151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10574.660280] Lustre: Skipped 1 previous similar message [10574.753561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10580.877941] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10581.145927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10581.153834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10581.199488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10588.902298] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10597.764402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10604.371271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10604.455887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10604.535640] Lustre: 444803:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10604.611057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10604.662742] Lustre: lustre-MDT0001: new disk, initializing [10604.811779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10604.878325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10604.892728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10608.038419] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10615.655815] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10615.842405] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10619.997527] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10620.246970] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10625.505731] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10625.509547] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10625.530167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10625.535903] Lustre: Skipped 1 previous similar message [10626.471709] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [10630.460392] LustreError: 448225:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10630.564960] Lustre: server umount lustre-MDT0001 complete [10635.750416] LustreError: 444811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.757989] LustreError: 444811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10640.867101] LustreError: 444810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.885701] LustreError: 444810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10645.995281] LustreError: 444815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10646.018108] LustreError: 444815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10648.040296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10648.044068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10648.050023] Lustre: Skipped 1 previous similar message [10648.086795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10651.108293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10651.112611] LustreError: 444815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10653.270110] LustreError: 448499:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [10653.274926] LustreError: 448499:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10653.496246] Lustre: server umount lustre-OST0000 complete [10656.227610] LustreError: 444815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10663.014691] LustreError: 448700:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [10663.026460] LustreError: 448700:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10663.346031] Lustre: server umount lustre-MDT0000 complete [10676.264590] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10678.628055] Key type lgssc unregistered [10678.920037] LNet: 449385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10678.934197] LNetError: 449385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10678.952665] LNet: Removed LNI 192.168.201.105@tcp [10679.825315] Key type .llcrypt unregistered [10679.830428] Key type ._llcrypt unregistered [10719.408289] Key type ._llcrypt registered [10719.416937] Key type .llcrypt registered [10719.544667] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10719.741505] Key type .llcrypt unregistered [10719.743194] Key type ._llcrypt unregistered [10735.574987] Key type ._llcrypt registered [10735.583656] Key type .llcrypt registered [10735.678792] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10736.715714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10736.846143] alg: No test for adler32 (adler32-zlib) [10737.844226] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [10738.058200] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10739.697887] Key type lgssc registered [10740.807417] Lustre: Echo OBD driver; http://www.lustre.org/ [10748.746518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10755.309088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10761.757957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10767.831825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10767.879709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10767.895943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.088363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10769.121975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10769.199751] Lustre: lustre-MDT0000: new disk, initializing [10769.320402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10769.344692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10773.069307] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10783.330191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10783.396691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10783.527619] Lustre: 452733:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10783.595622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10783.598687] Lustre: Skipped 1 previous similar message [10783.655754] Lustre: lustre-MDT0001: new disk, initializing [10783.746939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10783.783713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10783.795811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10787.815539] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10794.919206] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10798.981872] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10806.177023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10806.239517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10806.459615] Lustre: lustre-OST0000: new disk, initializing [10806.473532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10806.600529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10811.805556] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10815.031313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10815.040428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10815.126410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10817.877696] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10824.679709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10824.693662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10824.703815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.699900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.700204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.708827] Lustre: Skipped 1 previous similar message [10825.712556] Lustre: Skipped 1 previous similar message [10829.262421] LustreError: 455176:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10829.383665] Lustre: server umount lustre-MDT0000 complete [10833.193403] LustreError: 452725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764801738 with bad export cookie 4171272662962154510 [10833.196112] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10833.208759] LustreError: 452725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10833.464932] LustreError: 455379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10833.468222] LustreError: 455379:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10833.696345] Lustre: server umount lustre-MDT0001 complete [10847.252536] LustreError: 455580:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10847.268985] LustreError: 455580:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10847.513061] Lustre: server umount lustre-OST0000 complete [10858.925176] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:42:42 (1764801762) [10860.113652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10866.874235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10867.341051] LustreError: 456409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.366805] LustreError: 456409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10867.446841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10871.176986] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10872.808720] LustreError: 456409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.929674] LustreError: 456409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10879.086338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10879.418983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10882.653758] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10888.013400] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10891.983765] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10898.256326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10898.324837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10898.574215] Lustre: Found index 0 for lustre-OST0000, updating log [10898.788411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10904.578041] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10909.433767] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10913.142856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10920.418496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10920.427116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.445500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10920.448217] Lustre: Skipped 1 previous similar message [10927.595421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10927.602315] Lustre: Skipped 4 previous similar messages [10933.216199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10933.270698] LustreError: 458668:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10933.283367] LustreError: 458668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10933.578655] Lustre: server umount lustre-OST0000 complete [10936.211253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10938.856962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10938.869730] Lustre: Skipped 1 previous similar message [10938.875192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10949.088981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10949.098792] Lustre: Skipped 4 previous similar messages [10952.160191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10952.240976] LustreError: 458925:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10952.246879] LustreError: 458925:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10952.310185] Lustre: server umount lustre-MDT0000 complete [10954.209823] LustreError: 458319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.220527] LustreError: 458319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10955.559326] LustreError: 456388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764801861 with bad export cookie 4171272662962155903 [10955.560520] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10955.587930] LustreError: 456388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10956.101771] Lustre: server umount lustre-MDT0001 complete [10964.073585] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:44:28 (1764801868) [10965.149080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10965.906882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10966.793977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10967.762887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10968.681646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10969.647291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10970.606648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10971.864406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10972.991781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10974.046484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10975.018851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10976.183480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10977.389918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10978.485557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10979.590081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10980.658190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10981.905118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10983.127442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10984.388962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10985.494414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10986.470888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10987.541990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10988.492334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10989.404912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10990.341578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10991.314431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10992.423436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10993.336673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10994.506644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10995.621950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10996.594121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10997.501944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10998.331763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10999.168224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10999.963629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11000.705289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11001.507153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11002.483953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11003.438156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11004.364261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11005.462839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11006.469396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11007.355765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11008.254391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11009.227755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11010.204402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11011.058041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11012.032142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11012.887665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11013.886951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11014.902400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11022.814560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11023.196549] LustreError: 462529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11023.281985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11027.309453] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11032.547716] LustreError: 462528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.558260] LustreError: 462528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11034.753812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11038.308118] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11043.661209] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11047.548436] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11055.464964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11055.557851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11055.691970] Lustre: Found index 0 for lustre-OST0000, updating log [11056.038056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11056.041391] Lustre: Skipped 1 previous similar message [11063.099125] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11069.383568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11069.466626] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11072.483901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11072.487463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11072.494742] Lustre: Skipped 1 previous similar message [11072.532674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11077.440655] LustreError: 464794:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11077.447338] LustreError: 464794:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [11077.571793] Lustre: server umount lustre-OST0000 complete [11081.185043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11081.196994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11081.203231] Lustre: Skipped 1 previous similar message [11087.313083] Lustre: server umount lustre-MDT0000 complete [11087.336106] LustreError: 463592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.357094] LustreError: 463592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11090.562593] LustreError: 462509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764801996 with bad export cookie 4171272662962156988 [11090.565631] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11090.574284] LustreError: 462509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11090.889797] Lustre: server umount lustre-MDT0001 complete [11093.180747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11107.980756] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11117.107958] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11124.881325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11130.641191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11136.494620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11142.963731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11148.818706] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:47:33 (1764802053) [11161.988138] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11164.572812] Key type lgssc unregistered [11164.815227] LNet: 468766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11164.820784] LNetError: 468766:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11187.366711] LNet: Removed LNI 192.168.201.105@tcp [11188.195481] Key type .llcrypt unregistered [11188.205657] Key type ._llcrypt unregistered [11202.671532] Key type ._llcrypt registered [11202.673132] Key type .llcrypt registered [11202.791880] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11203.645834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11203.695707] alg: No test for adler32 (adler32-zlib) [11204.703823] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [11204.918357] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11206.544464] Key type lgssc registered [11207.614609] Lustre: Echo OBD driver; http://www.lustre.org/ [11211.507142] LNet: Added LNI 192.168.201.105@tcp100 [8/256/0/180] [11213.001525] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [11226.152897] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11228.367176] Key type lgssc unregistered [11228.652987] LNet: 470639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11228.658900] LNetError: 470639:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11228.662835] LNet: Removed LNI 192.168.201.105@tcp [11228.680950] LNet: Removed LNI 192.168.201.105@tcp100 [11229.181339] Key type .llcrypt unregistered [11229.183194] Key type ._llcrypt unregistered [11249.219809] Key type ._llcrypt registered [11249.221228] Key type .llcrypt registered [11249.305949] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11261.467215] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11262.274865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11262.402588] alg: No test for adler32 (adler32-zlib) [11263.399403] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [11263.570972] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11265.208184] Key type lgssc registered [11266.018030] Lustre: Echo OBD driver; http://www.lustre.org/ [11273.590042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11279.903731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11286.724929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11294.105332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11301.975380] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 17:50:05 (1764802205) [11317.223604] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11319.965514] Key type lgssc unregistered [11320.201550] LNet: 474659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11320.206908] LNetError: 474659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11320.218670] LNet: Removed LNI 192.168.201.105@tcp [11320.988497] Key type .llcrypt unregistered [11320.990684] Key type ._llcrypt unregistered [11335.914233] Key type ._llcrypt registered [11335.920264] Key type .llcrypt registered [11336.027337] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11336.769889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11336.800734] alg: No test for adler32 (adler32-zlib) [11337.806806] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [11337.984748] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11339.632280] Key type lgssc registered [11340.520418] Lustre: Echo OBD driver; http://www.lustre.org/ [11349.163969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11349.211093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11349.226571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.449634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11350.485781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11350.554284] Lustre: lustre-MDT0000: new disk, initializing [11350.643357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11350.670528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11354.198257] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11363.102225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11363.195561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11363.271773] Lustre: 475856:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11363.349701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11363.353479] Lustre: Skipped 1 previous similar message [11363.410854] Lustre: lustre-MDT0001: new disk, initializing [11363.503082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11363.545138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11363.567496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11367.067771] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11373.060221] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11375.830827] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11382.180387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11382.243126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11382.429532] Lustre: lustre-OST0000: new disk, initializing [11382.433289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11382.515342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11388.750726] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11389.026759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11389.033671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11389.157099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11395.767897] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11398.878555] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing lnet_if_list [11403.088497] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11403.117900] LNet: Added LNI 192.168.201.105@tcp42 [8/256/0/180] [11420.131074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11420.133876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.139284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11420.147034] Lustre: Skipped 1 previous similar message [11422.782215] LustreError: 479237:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11422.913820] Lustre: server umount lustre-OST0000 complete [11430.370858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11430.372499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11430.372937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11430.376034] Lustre: Skipped 1 previous similar message [11430.379223] Lustre: Skipped 1 previous similar message [11435.490545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11435.493771] Lustre: Skipped 1 previous similar message [11439.589108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11439.595229] Lustre: Skipped 1 previous similar message [11440.096148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11440.173641] LustreError: 479438:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11440.183376] LustreError: 479438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11440.228546] Lustre: server umount lustre-MDT0000 complete [11443.296946] LustreError: 475847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764802349 with bad export cookie 6667326087296466543 [11443.300989] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11443.312180] LustreError: 475847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11443.449132] LustreError: 479644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11443.452271] LustreError: 479644:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11443.631375] Lustre: server umount lustre-MDT0001 complete [11450.491121] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11452.306854] Key type lgssc unregistered [11452.574470] LNet: 480185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11452.579185] LNetError: 480185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11452.583442] LNet: Removed LNI 192.168.201.105@tcp [11452.593389] LNet: Removed LNI 192.168.201.105@tcp42 [11453.172744] Key type .llcrypt unregistered [11453.175590] Key type ._llcrypt unregistered [11468.105306] Key type ._llcrypt registered [11468.106961] Key type .llcrypt registered [11468.180528] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11468.676970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11468.688405] alg: No test for adler32 (adler32-zlib) [11469.680227] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [11469.825051] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11471.472189] Key type lgssc registered [11472.247718] Lustre: Echo OBD driver; http://www.lustre.org/ [11477.808819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11477.823048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11479.214892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11482.255363] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11488.738398] LustreError: 481409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11488.746507] LustreError: 481409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11489.431164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11489.742231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11493.498481] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11497.882884] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11500.678976] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11507.312568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11507.725660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11513.221428] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11519.486829] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11535.073658] loop0: detected capacity change from 0 to 204800000 [11535.293860] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11535.305411] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11535.319899] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11535.379511] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11541.846215] loop0: detected capacity change from 0 to 204800000 [11542.131153] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11542.140802] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11542.153398] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11542.305379] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11547.145895] loop0: detected capacity change from 0 to 204800000 [11552.376486] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11552.479335] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11552.635670] Lustre: fs15246-OST0000: new disk, initializing [11552.651989] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11552.703350] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11558.053658] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11567.747308] loop1: detected capacity change from 0 to 204800000 [11572.801515] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11572.874835] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11572.935214] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11572.971430] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11573.028585] Lustre: fs15246-MDT0000: new disk, initializing [11573.225577] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11573.244636] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11573.369112] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11574.434475] Lustre: 485781:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@0000000053602527 x1850529415213696/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1764802496 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11576.549590] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11579.495831] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11579.501927] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11579.538654] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11593.702214] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11593.714567] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11598.829547] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11599.651223] LustreError: 486909:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [11599.735270] Lustre: server umount fs15246-MDT0000 complete [11603.941094] LustreError: 481403:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.108142] LustreError: 487411:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [11608.117758] LustreError: 487411:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11608.263825] Lustre: server umount fs15246-OST0000 complete [11619.307720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11619.310139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11619.317159] Lustre: Skipped 1 previous similar message [11624.419857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11624.428157] Lustre: Skipped 2 previous similar messages [11625.034583] LustreError: 487912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11625.040207] LustreError: 487912:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [11625.181636] Lustre: server umount lustre-OST0000 complete [11629.543065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11629.547242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11629.561371] Lustre: Skipped 1 previous similar message [11639.779826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11639.782595] Lustre: Skipped 3 previous similar messages [11643.360270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11643.439084] LustreError: 488114:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11643.442168] LustreError: 488114:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11643.499298] Lustre: server umount lustre-MDT0000 complete [11644.900539] LustreError: 481408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11644.912845] LustreError: 481408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11646.979287] LustreError: 481388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764802552 with bad export cookie 8178056036387404350 [11646.991627] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11646.994544] LustreError: 481388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11647.423676] Lustre: server umount lustre-MDT0001 complete [11658.508214] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11661.298747] Key type lgssc unregistered [11661.554116] LNet: 488850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11661.563622] LNetError: 488850:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11661.595971] LNet: Removed LNI 192.168.201.105@tcp [11662.286258] Key type .llcrypt unregistered [11662.292280] Key type ._llcrypt unregistered [11680.542394] Key type ._llcrypt registered [11680.544206] Key type .llcrypt registered [11680.627293] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11681.468297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11681.514076] alg: No test for adler32 (adler32-zlib) [11682.553850] Lustre: Lustre: Build Version: 2.16.59_23_g5b5e124 [11682.863880] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11684.561206] Key type lgssc registered [11685.878287] Lustre: Echo OBD driver; http://www.lustre.org/ [11693.912113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11698.799219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11703.725213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11708.888285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11723.191342] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11732.511362] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11742.253820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11748.985753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11754.768525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11761.766342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11768.615529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11768.668239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11768.687622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11769.891478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11769.921777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11769.984737] Lustre: lustre-MDT0000: new disk, initializing [11770.045288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11770.066100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11773.403753] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11783.837314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11783.931971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11784.036529] Lustre: 493286:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11784.113354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11784.117053] Lustre: Skipped 1 previous similar message [11784.194706] Lustre: lustre-MDT0001: new disk, initializing [11784.319308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11784.372841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11784.390605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11788.299265] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11795.234141] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11798.520522] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11804.279617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11804.358922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11804.644126] Lustre: lustre-OST0000: new disk, initializing [11804.647927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11804.747322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11809.293736] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11814.450795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11814.457577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11814.553055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11815.651608] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11819.513285] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11819.721445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11823.360094] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11823.543740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11825.127127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11825.130876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11825.139308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11830.249246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11830.253822] Lustre: Skipped 2 previous similar messages [11831.185963] LustreError: 496226:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11831.461994] Lustre: server umount lustre-OST0000 complete [11835.372719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.375165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11835.382250] Lustre: Skipped 1 previous similar message [11835.383918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11835.385551] LustreError: Skipped 1 previous similar message [11835.396359] Lustre: Skipped 1 previous similar message [11840.481807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11840.484124] Lustre: Skipped 1 previous similar message [11841.304167] LustreError: 496427:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11841.315786] LustreError: 496427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11841.507678] Lustre: server umount lustre-MDT0000 complete [11845.162946] LustreError: 493277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764802750 with bad export cookie 13267538717039284929 [11845.171627] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11845.183465] LustreError: 493277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11845.382978] LustreError: 496629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11845.397859] LustreError: 496629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11845.711547] Lustre: server umount lustre-MDT0001 complete [11869.506621] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11877.555636] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11885.524372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11891.107796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11896.223310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11902.587449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11904.171127] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11754 sec ========= 18:00:08 (1764802808) [11906.179648] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:00:10 (1764802810) === [11909.803937] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:00:13 (1764802813) === [11942.154479] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11944.552358] Key type lgssc unregistered [11944.820401] LNet: 501488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11944.829954] LNetError: 501488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11944.851731] LNet: Removed LNI 192.168.201.105@tcp [11945.560285] Key type .llcrypt unregistered [11945.561927] Key type ._llcrypt unregistered