[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486925270 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002012] Switched APIC routing to physical x2apic. [ 0.003018] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007017] pid_max: default: 32768 minimum: 301 [ 0.008141] LSM: Security Framework initializing [ 0.009086] Yama: becoming mindful. [ 0.010050] SELinux: Initializing. [ 0.011073] *** VALIDATE selinux *** [ 0.021416] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024146] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028103] *** VALIDATE tmpfs *** [ 0.030110] *** VALIDATE proc *** [ 0.031326] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033320] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036021] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041005] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.043163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044762] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048016] ... value mask: 0000ffffffffffff [ 0.049021] ... max period: 00007fffffffffff [ 0.050025] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052344] rcu: Hierarchical SRCU implementation. [ 0.054568] smp: Bringing up secondary CPUs ... [ 0.055597] x86: Booting SMP configuration: [ 0.056029] .... node #0, CPUs: #1 #2 #3 [ 0.059602] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141021] node 0 deferred pages initialised in 76ms [ 0.145017] devtmpfs: initialized [ 0.146224] x86/mm: Memory block size: 128MB [ 0.148744] gcov: version magic: 0x41383552 [ 0.149634] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153138] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156456] pinctrl core: initialized pinctrl subsystem [ 0.158244] [ 0.158635] ************************************************************* [ 0.161018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163020] ** ** [ 0.165026] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168026] ** ** [ 0.170018] ** This means that this kernel is built to expose internal ** [ 0.172021] ** IOMMU data structures, which may compromise security on ** [ 0.173015] ** your system. ** [ 0.176024] ** ** [ 0.178020] ** If you see this message and you are not debugging the ** [ 0.180023] ** kernel, report this immediately to your vendor! ** [ 0.183022] ** ** [ 0.185020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187022] ************************************************************* [ 0.189896] NET: Registered protocol family 16 [ 0.192534] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201165] cpuidle: using governor menu [ 0.202945] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205018] PCI: Using configuration type 1 for base access [ 0.207151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224168] cryptd: max_cpu_qlen set to 1000 [ 0.226079] ACPI: Added _OSI(Module Device) [ 0.227532] ACPI: Added _OSI(Processor Device) [ 0.228016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233714] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240379] ACPI: Interpreter enabled [ 0.242049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243032] ACPI: Using IOAPIC for interrupt routing [ 0.245170] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248469] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257869] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262000] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.264476] acpiphp: Slot [2] registered [ 0.267150] acpiphp: Slot [5] registered [ 0.268151] acpiphp: Slot [6] registered [ 0.270166] acpiphp: Slot [7] registered [ 0.271139] acpiphp: Slot [8] registered [ 0.272196] acpiphp: Slot [9] registered [ 0.274175] acpiphp: Slot [10] registered [ 0.276174] acpiphp: Slot [3] registered [ 0.277138] acpiphp: Slot [4] registered [ 0.278100] acpiphp: Slot [11] registered [ 0.280122] acpiphp: Slot [12] registered [ 0.282172] acpiphp: Slot [13] registered [ 0.283143] acpiphp: Slot [14] registered [ 0.284142] acpiphp: Slot [15] registered [ 0.286126] acpiphp: Slot [16] registered [ 0.287130] acpiphp: Slot [17] registered [ 0.288177] acpiphp: Slot [18] registered [ 0.290169] acpiphp: Slot [19] registered [ 0.292231] acpiphp: Slot [20] registered [ 0.294140] acpiphp: Slot [21] registered [ 0.295106] acpiphp: Slot [22] registered [ 0.296114] acpiphp: Slot [23] registered [ 0.298106] acpiphp: Slot [24] registered [ 0.300137] acpiphp: Slot [25] registered [ 0.302149] acpiphp: Slot [26] registered [ 0.303116] acpiphp: Slot [27] registered [ 0.305125] acpiphp: Slot [28] registered [ 0.307122] acpiphp: Slot [29] registered [ 0.308115] acpiphp: Slot [30] registered [ 0.310123] acpiphp: Slot [31] registered [ 0.312071] PCI host bridge to bus 0000:00 [ 0.314029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.321025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.323023] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.326048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.328276] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334479] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.345022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.350487] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.353030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.355025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.360739] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.363829] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.369827] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.389020] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.396015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402449] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.409023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430025] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.442796] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.457027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475022] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.484336] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.495033] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.510041] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.519867] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.525017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.545022] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.555960] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.563024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.570023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.586016] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.597411] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.622017] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.633061] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.634400] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.637362] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640408] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.643254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.648206] iommu: Default domain type: Passthrough [ 0.649480] SCSI subsystem initialized [ 0.650000] ACPI: bus type USB registered [ 0.652158] usbcore: registered new interface driver usbfs [ 0.655094] usbcore: registered new interface driver hub [ 0.657122] usbcore: registered new device driver usb [ 0.659184] pps_core: LinuxPPS API ver. 1 registered [ 0.661014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.664056] PTP clock support registered [ 0.666344] EDAC MC: Ver: 3.0.0 [ 0.668131] PCI: Using ACPI for IRQ routing [ 0.670418] NetLabel: Initializing [ 0.672012] NetLabel: domain hash size = 128 [ 0.673013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676081] NetLabel: unlabeled traffic allowed by default [ 0.678127] vgaarb: loaded [ 0.680256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.682017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.689027] clocksource: Switched to clocksource kvm-clock [ 0.795263] VFS: Disk quotas dquot_6.6.0 [ 0.796795] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.799629] *** VALIDATE ramfs *** [ 0.801255] *** VALIDATE hugetlbfs *** [ 0.802986] pnp: PnP ACPI init [ 0.805458] pnp: PnP ACPI: found 6 devices [ 0.823322] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.827300] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.829843] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.832719] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.835693] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.838331] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.841650] NET: Registered protocol family 2 [ 0.844286] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.849239] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.853180] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.858766] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.862616] TCP: Hash tables configured (established 65536 bind 65536) [ 0.866365] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.869601] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873286] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.876328] NET: Registered protocol family 1 [ 0.879299] RPC: Registered named UNIX socket transport module. [ 0.882616] RPC: Registered udp transport module. [ 0.884488] RPC: Registered tcp transport module. [ 0.886182] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.888927] NET: Registered protocol family 44 [ 0.890488] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.893043] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.895455] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.898312] PCI: CLS 0 bytes, default 64 [ 0.901416] Unpacking initramfs... [ 2.313298] debug: unmapping init [mem 0xffff95b23cc54000-0xffff95b23ffbffff] [ 2.317477] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.319625] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.322637] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.825054] Initialise system trusted keyrings [ 2.826733] Key type blacklist registered [ 2.828686] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.838124] zbud: loaded [ 2.841127] *** VALIDATE nfs *** [ 2.842300] *** VALIDATE nfs4 *** [ 2.844236] pstore: using deflate compression [ 2.847052] Platform Keyring initialized [ 2.964819] NET: Registered protocol family 38 [ 2.967271] Key type asymmetric registered [ 2.968853] Asymmetric key parser 'x509' registered [ 2.970968] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.974189] io scheduler mq-deadline registered [ 2.975844] io scheduler kyber registered [ 2.977687] io scheduler bfq registered [ 2.979900] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.983362] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.986893] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.989738] ACPI: Power Button [PWRF] [ 3.080843] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.172864] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.352023] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.445299] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.626279] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.655595] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.688227] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.697938] Non-volatile memory driver v1.3 [ 3.699798] Linux agpgart interface v0.103 [ 3.735222] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.738236] vda: detected capacity change from 0 to 68526080 [ 3.756701] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.759841] vdb: detected capacity change from 0 to 1073741824 [ 3.777449] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.781711] vdc: detected capacity change from 0 to 2621440000 [ 3.797860] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.801173] vdd: detected capacity change from 0 to 2621440000 [ 3.816521] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.819560] vde: detected capacity change from 0 to 4294967296 [ 3.836782] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.839732] vdf: detected capacity change from 0 to 4294967296 [ 3.846430] libphy: Fixed MDIO Bus: probed [ 3.856609] usbcore: registered new interface driver usbserial_generic [ 3.859353] usbserial: USB Serial support registered for generic [ 3.861908] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.866441] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.868216] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.870800] mousedev: PS/2 mouse device common for all mice [ 3.874328] rtc_cmos 00:05: RTC can wake from S4 [ 3.876965] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.879955] rtc_cmos 00:05: registered as rtc0 [ 3.883338] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.884737] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.886900] intel_pstate: CPU model not supported [ 3.895700] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.899563] hid: raw HID events driver (C) Jiri Kosina [ 3.901531] usbcore: registered new interface driver usbhid [ 3.904313] usbhid: USB HID core driver [ 3.907046] drop_monitor: Initializing network drop monitor service [ 3.909722] Initializing XFRM netlink socket [ 3.911936] NET: Registered protocol family 10 [ 3.914835] Segment Routing with IPv6 [ 3.916384] NET: Registered protocol family 17 [ 3.918730] mpls_gso: MPLS GSO support [ 3.926292] RAS: Correctable Errors collector initialized. [ 3.928806] AVX version of gcm_enc/dec engaged. [ 3.931545] AES CTR mode by8 optimization enabled [ 4.006314] sched_clock: Marking stable (4006288768, 0)->(4966085458, -959796690) [ 4.010357] registered taskstats version 1 [ 4.012151] Loading compiled-in X.509 certificates [ 4.014097] zswap: loaded using pool lzo/zbud [ 4.037295] Key type big_key registered [ 4.048837] Key type encrypted registered [ 4.050378] ima: No TPM chip found, activating TPM-bypass! [ 4.052208] ima: Allocated hash algorithm: sha1 [ 4.053750] ima: No architecture policies found [ 4.055416] evm: Initialising EVM extended attributes: [ 4.057149] evm: security.selinux [ 4.058316] evm: security.ima [ 4.059308] evm: security.capability [ 4.060448] evm: HMAC attrs: 0x1 [ 4.062703] rtc_cmos 00:05: setting system clock to 2025-11-17 03:18:19 UTC (1763349499) [ 4.068643] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 4.071391] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 4.080139] Write protecting the kernel read-only data: 28672k [ 4.083366] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 4.085974] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 4.116937] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.125021] systemd[1]: Detected virtualization kvm. [ 4.126698] systemd[1]: Detected architecture x86-64. [ 4.128437] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.153407] systemd[1]: No hostname configured. [ 4.155390] systemd[1]: Set hostname to . [ 4.157647] random: systemd: uninitialized urandom read (16 bytes read) [ 4.160307] systemd[1]: Initializing machine ID from random generator. [ 4.208528] random: ln: uninitialized urandom read (6 bytes read) [ 4.310981] random: systemd: uninitialized urandom read (16 bytes read) [ 4.313620] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.317606] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.321965] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.021721] device-mapper: uevent: version 1.0.3 [ 5.023903] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.768509] random: fast init done [ 5.774459] virtio_net virtio0 ens2: renamed from eth0 [ 5.872692] scsi host0: ata_piix [ 5.878946] scsi host1: ata_piix [ 5.880297] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.881793] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.590370] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.564885] random: crng init done [ 10.566256] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.062159] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.216459] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.463489] SELinux: Disabled at runtime. [ 12.518972] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.528706] systemd[1]: Detected virtualization kvm. [ 12.530631] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.976550] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.980369] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.986174] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.992238] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.995513] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.003804] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.008048] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 13.095840] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK [[ 13.428577] squashfs: version 4.0 (2009/01/31) Phillip Lougher 0m] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.762708] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.766993] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.925156] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.934216] EDAC sbridge: Ver: 1.1.2 [ 15.474421] Key type dns_resolver registered [ 15.797312] NFS: Registering the id_resolver key type [ 15.799400] Key type id_resolver registered [ 15.801719] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg116-server login: [ 58.686981] libcfs: loading out-of-tree module taints kernel. [ 58.715448] Key type ._llcrypt registered [ 58.717142] Key type .llcrypt registered [ 58.813441] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 60.446902] hrtimer: interrupt took 5757877 ns [ 74.752580] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 76.053684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.064798] alg: No test for adler32 (adler32-zlib) [ 77.305469] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 78.049271] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 79.800219] Key type lgssc registered [ 81.667464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.317275] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.860885] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.531358] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.669796] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.090221] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.655919] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 141.832856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.919190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.960923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.296704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.336785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.401781] Lustre: lustre-MDT0000: new disk, initializing [ 143.508559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.542524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.632765] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.906133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.998347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 158.114645] Lustre: 6530:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.160487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.163417] Lustre: Skipped 1 previous similar message [ 158.226786] Lustre: lustre-MDT0001: new disk, initializing [ 158.301367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.333837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.348096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.554301] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.453910] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 173.452653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.515295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 173.691436] Lustre: lustre-OST0000: new disk, initializing [ 173.710714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 173.774802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.045647] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.187831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.210236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.288981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.021136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.094418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 190.181478] Lustre: lustre-OST0001: new disk, initializing [ 190.184949] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 190.243958] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.613273] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.577854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 195.582132] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 195.664602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 203.960731] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 209.387083] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 218.938494] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing check_logdir /tmp/testlogs/ [ 222.508480] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing yml_node [ 225.406728] Lustre: DEBUG MARKER: Client: 2.16.61.42 [ 227.187135] Lustre: DEBUG MARKER: MDS: 2.16.61.42 [ 229.642205] Lustre: DEBUG MARKER: OSS: 2.16.61.42 [ 231.365557] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 22:22:05 EST 2025 [ 244.833595] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 245.939513] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 260.579122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 260.586911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.597200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.118072] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.118714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.131692] Lustre: Skipped 2 previous similar messages [ 262.143060] Lustre: Skipped 3 previous similar messages [ 266.052338] LustreError: 12239:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 266.121495] Lustre: server umount lustre-MDT0000 complete [ 268.924107] LustreError: 6523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349764 with bad export cookie 12882510829818284274 [ 268.925518] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 268.936562] LustreError: 6523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 272.353605] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.353851] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.356683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 272.359059] Lustre: Skipped 1 previous similar message [ 272.365682] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 273.380196] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 273.398449] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 275.170555] LustreError: 12441:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 275.174491] LustreError: 12441:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 275.299373] Lustre: server umount lustre-MDT0001 complete [ 279.069224] LustreError: 12642:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 279.084905] LustreError: 12642:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 279.204896] Lustre: server umount lustre-OST0000 complete [ 283.241045] LustreError: 12843:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 283.251146] LustreError: 12843:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 283.411051] Lustre: server umount lustre-OST0001 complete [ 288.437938] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 295.610526] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 302.565995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 307.772724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 313.130288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.563907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 324.063824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 324.107476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.275586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 324.327094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 324.401816] Lustre: lustre-MDT0000: new disk, initializing [ 324.478422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 324.494979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 327.498474] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 335.201637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.257485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 335.312256] Lustre: 15025:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 335.355266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 335.358169] Lustre: Skipped 1 previous similar message [ 335.408794] Lustre: lustre-MDT0001: new disk, initializing [ 335.463181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 335.488103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 335.494345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 338.159210] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 344.412801] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 347.836875] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 355.021101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 355.122294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 355.486138] Lustre: lustre-OST0000: new disk, initializing [ 355.493634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 355.581865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.184997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 357.195779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 357.578763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 363.285826] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 372.355207] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 375.980798] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 376.158738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 379.549044] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 379.702239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 382.956244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 382.960455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.961393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 382.961400] Lustre: Skipped 3 previous similar messages [ 382.990016] Lustre: Skipped 1 previous similar message [ 387.644131] LustreError: 17967:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 387.655402] LustreError: 17967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 387.810071] Lustre: server umount lustre-OST0000 complete [ 391.649650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 391.661269] LustreError: Skipped 1 previous similar message [ 391.668241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.684955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 391.689275] Lustre: Skipped 1 previous similar message [ 397.496117] LustreError: 18168:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 397.499064] LustreError: 18168:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 397.606893] Lustre: server umount lustre-MDT0000 complete [ 398.309167] LustreError: 15897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.325482] LustreError: 15897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 400.709351] LustreError: 15816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349896 with bad export cookie 12882510829818285807 [ 400.716631] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 400.724279] LustreError: 15816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 400.942960] Lustre: server umount lustre-MDT0001 complete [ 404.727371] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:24:58 (1763349898) [ 406.987332] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 408.423381] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:25:02 (1763349902) [ 414.687414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.035572] LustreError: 19038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.096919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.726360] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 420.324233] LustreError: 19038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.827672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 429.467038] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 433.789976] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 436.807805] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 442.726075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 447.950246] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 452.325377] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 471.009087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 471.012740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.013499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.024390] Lustre: Skipped 2 previous similar messages [ 471.030709] Lustre: Skipped 3 previous similar messages [ 474.386157] LustreError: 21303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 474.390226] LustreError: 21303:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 474.491538] Lustre: server umount lustre-OST0000 complete [ 483.668827] Lustre: server umount lustre-MDT0000 complete [ 486.370913] LustreError: 19032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.392223] LustreError: 19032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 486.779504] LustreError: 19020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349982 with bad export cookie 12882510829818286682 [ 486.795320] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.998535] Lustre: server umount lustre-MDT0001 complete [ 496.026758] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 498.122243] Key type lgssc unregistered [ 498.402335] LNet: 22237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 498.411391] LNetError: 22237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 498.430606] LNet: Removed LNI 192.168.201.116@tcp [ 499.211165] Key type .llcrypt unregistered [ 499.213174] Key type ._llcrypt unregistered [ 518.318749] Key type ._llcrypt registered [ 518.320268] Key type .llcrypt registered [ 518.401283] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 519.226886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 519.345099] alg: No test for adler32 (adler32-zlib) [ 520.327077] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 520.550214] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 522.216176] Key type lgssc registered [ 523.244839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 531.441375] loop: module loaded [ 531.494306] loop0: detected capacity change from 0 to 204800000 [ 531.712406] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 531.732824] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 531.944513] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 531.981366] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 537.189420] loop0: detected capacity change from 0 to 204800000 [ 537.342941] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 537.351776] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 537.375512] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 537.514519] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 542.828385] loop0: detected capacity change from 0 to 204800000 [ 548.518814] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 548.565518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 548.583520] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.761037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 549.804787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 549.869969] Lustre: lustre-MDT0000: new disk, initializing [ 549.950182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.965321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 552.577625] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 559.645425] loop1: detected capacity change from 0 to 204800000 [ 564.219835] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 564.278031] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 564.521450] Lustre: lustre-OST07c6: new disk, initializing [ 564.532480] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 564.535094] Lustre: Skipped 1 previous similar message [ 564.623287] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 566.617823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 566.627395] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 566.674609] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 570.629331] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 575.429296] Lustre: 24387:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 575.439251] Lustre: 24387:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 575.450624] Lustre: 24387:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 575.458183] Lustre: 24387:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 575.469514] Lustre: 24387:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 575.475590] Lustre: 24387:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 595.144341] Lustre: 24389:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 595.157982] Lustre: 24389:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 595.164504] Lustre: 24389:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 595.170827] Lustre: 24389:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 595.181477] Lustre: 24389:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 595.191391] Lustre: 24389:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 599.497689] Lustre: 24388:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 599.509633] Lustre: 24388:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 599.516100] Lustre: 24388:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 599.526189] Lustre: 24388:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.531262] Lustre: 24388:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 599.546480] Lustre: 24388:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.554789] Lustre: 24388:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 599.570905] Lustre: 24388:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.580502] Lustre: 24388:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 599.596130] Lustre: 24388:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 599.610716] Lustre: 24388:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 599.615514] Lustre: 24388:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.774432] Lustre: 26054:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 602.785216] Lustre: 26054:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 602.802150] Lustre: 26054:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 602.806436] Lustre: 26054:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.812283] Lustre: 26054:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 602.817731] Lustre: 26054:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.826191] Lustre: 26054:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 602.830413] Lustre: 26054:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.835253] Lustre: 26054:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 602.843767] Lustre: 26054:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.854083] Lustre: 26054:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 602.858531] Lustre: 26054:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.310957] Lustre: 24388:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 607.320262] Lustre: 24388:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 607.335271] Lustre: 24388:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 607.347622] Lustre: 24388:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.352944] Lustre: 24388:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 607.360682] Lustre: 24388:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.364663] Lustre: 24388:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 607.369562] Lustre: 24388:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.374681] Lustre: 24388:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 607.386583] Lustre: 24388:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.398679] Lustre: 24388:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 607.407854] Lustre: 24388:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 618.289181] Lustre: 24388:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 618.308475] Lustre: 24388:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 618.326028] Lustre: 24388:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 618.338152] Lustre: 24388:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 618.352566] Lustre: 24388:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 618.361148] Lustre: 24388:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 618.372072] Lustre: 24388:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 618.383715] Lustre: 24388:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 618.389332] Lustre: 24388:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 618.395448] Lustre: 24388:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 618.400060] Lustre: 24388:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 618.403741] Lustre: 24388:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 669.154155] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.156285] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 669.184096] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 674.274903] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 678.368111] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 678.409088] LustreError: 26574:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 678.520449] Lustre: server umount lustre-OST07c6 complete [ 686.182426] LustreError: 27078:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 686.188900] LustreError: 27078:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 686.310029] Lustre: server umount lustre-MDT0000 complete [ 702.262758] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 704.716481] Key type lgssc unregistered [ 704.938307] LNet: 27914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.947742] LNetError: 27914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.959282] LNet: Removed LNI 192.168.201.116@tcp [ 706.635523] Key type .llcrypt unregistered [ 706.639772] Key type ._llcrypt unregistered [ 722.079627] Key type ._llcrypt registered [ 722.081653] Key type .llcrypt registered [ 722.165267] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 722.879333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 722.888906] alg: No test for adler32 (adler32-zlib) [ 723.880458] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 724.045601] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 725.672599] Key type lgssc registered [ 726.698208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 733.447140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 733.464115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.860621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.250526] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 745.443537] LustreError: 29157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.464559] LustreError: 29157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 746.733176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 747.032204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 750.203313] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 754.286277] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 757.181584] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 763.398090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 763.845298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.200322] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 774.846330] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 780.213671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 792.546196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.549175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.571428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 795.622753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 795.629537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 795.633982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.732102] LustreError: 31608:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 796.820977] Lustre: server umount lustre-OST0000 complete [ 800.740127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.782687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 800.790592] Lustre: Skipped 2 previous similar messages [ 805.858224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 806.556829] LustreError: 31809:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 806.559609] LustreError: 31809:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 806.627910] Lustre: server umount lustre-MDT0000 complete [ 810.586080] LustreError: 29139:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350306 with bad export cookie 15167273339090852795 [ 810.587209] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.598066] LustreError: 29139:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 810.735823] LustreError: 32013:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 810.738588] LustreError: 32013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 811.000769] Lustre: server umount lustre-MDT0001 complete [ 822.598503] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 824.722908] Key type lgssc unregistered [ 824.997482] LNet: 32545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.006920] LNetError: 32545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.028738] LNet: Removed LNI 192.168.201.116@tcp [ 825.806372] Key type .llcrypt unregistered [ 825.810910] Key type ._llcrypt unregistered [ 839.801177] Key type ._llcrypt registered [ 839.802734] Key type .llcrypt registered [ 839.926519] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 840.459317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 840.471445] alg: No test for adler32 (adler32-zlib) [ 841.471512] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 841.674299] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 843.312173] Key type lgssc registered [ 844.442646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.361370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.384070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.898898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.873884] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 861.753614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 862.160392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.480359] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 869.663724] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 872.808092] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 878.591247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 878.928094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.702978] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 888.845295] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 898.413452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 908.770886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.773380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 908.778579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.787358] Lustre: Skipped 1 previous similar message [ 913.893659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.899667] Lustre: Skipped 1 previous similar message [ 914.251370] LustreError: 36047:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 914.391646] Lustre: server umount lustre-OST0000 complete [ 918.497090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 918.512890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.541622] Lustre: Skipped 1 previous similar message [ 918.555911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.752029] LustreError: 36249:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 923.754820] LustreError: 36249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 923.838756] Lustre: server umount lustre-MDT0000 complete [ 924.130358] LustreError: 35880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.144598] LustreError: 35880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 927.003887] LustreError: 33755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350422 with bad export cookie 15690751769120525444 [ 927.007327] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.035014] LustreError: 33755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 927.202859] LustreError: 36450:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 927.205707] LustreError: 36450:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 927.399948] Lustre: server umount lustre-MDT0001 complete [ 938.322693] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 941.091260] Key type lgssc unregistered [ 941.339782] LNet: 36982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.356214] LNetError: 36982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.390900] LNet: Removed LNI 192.168.201.116@tcp [ 942.137485] Key type .llcrypt unregistered [ 942.139420] Key type ._llcrypt unregistered [ 964.041106] Key type ._llcrypt registered [ 964.042617] Key type .llcrypt registered [ 964.171810] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 964.921518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.928891] alg: No test for adler32 (adler32-zlib) [ 966.031595] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 966.267781] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 967.920238] Key type lgssc registered [ 968.974103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 976.520645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 976.538371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.993457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.825385] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 988.134368] LustreError: 38357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.156834] LustreError: 38357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 989.548537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 989.934104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 993.011143] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 997.482486] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1000.251561] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1005.401407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1005.680972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.363189] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1015.764415] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1023.098395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1033.009434] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1033.706168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.706989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.715606] Lustre: Skipped 2 previous similar messages [ 1033.724272] Lustre: Skipped 1 previous similar message [ 1037.986132] LustreError: 40629:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1038.061277] Lustre: server umount lustre-MDT0000 complete [ 1038.089376] LustreError: 40470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1040.845342] LustreError: 38339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350536 with bad export cookie 13561919626581271121 [ 1040.848136] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.856635] LustreError: 38339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1043.217611] LustreError: 40470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.235770] LustreError: 40470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1043.248507] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 1043.252899] Lustre: Skipped 1 previous similar message [ 1043.942160] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.985043] LustreError: 39977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.985297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1046.004441] LustreError: 39977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1046.013015] Lustre: Skipped 1 previous similar message [ 1051.107952] LustreError: 39977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.113161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1051.129112] LustreError: 39977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1051.131689] Lustre: Skipped 1 previous similar message [ 1055.201164] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1055.294959] LustreError: 40830:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1055.297647] LustreError: 40830:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1055.398356] Lustre: server umount lustre-MDT0001 complete [ 1070.014572] LustreError: 41055:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1070.018889] LustreError: 41055:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1070.142351] Lustre: server umount lustre-OST0000 complete [ 1082.838990] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1085.003128] Key type lgssc unregistered [ 1085.231251] LNet: 41892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.238157] LNetError: 41892:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.254892] LNet: Removed LNI 192.168.201.116@tcp [ 1085.938446] Key type .llcrypt unregistered [ 1085.942688] Key type ._llcrypt unregistered [ 1106.403046] Key type ._llcrypt registered [ 1106.404696] Key type .llcrypt registered [ 1106.500161] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1107.254295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1107.296070] alg: No test for adler32 (adler32-zlib) [ 1108.392640] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 1108.601651] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1110.264128] Key type lgssc registered [ 1111.214704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1118.465733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1118.501450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.103235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.138604] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1130.466560] LustreError: 43290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1130.479180] LustreError: 43290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1132.512345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1132.888958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1137.776986] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1143.966959] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1147.938938] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1155.804930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1156.157540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.604194] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1165.378216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1167.992660] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1175.523897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1175.529193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.535081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1175.557667] Lustre: Skipped 1 previous similar message [ 1175.576157] Lustre: Skipped 1 previous similar message [ 1180.345539] LustreError: 45538:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1180.443104] Lustre: server umount lustre-OST0000 complete [ 1199.587445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1199.593574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.602841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.605650] Lustre: Skipped 1 previous similar message [ 1201.123071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.124652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.528557] LustreError: 45914:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1202.532395] LustreError: 45914:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1202.594360] Lustre: server umount lustre-MDT0000 complete [ 1205.820778] LustreError: 43271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350701 with bad export cookie 14020236581102957669 [ 1205.821833] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1205.829924] LustreError: 43271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1212.043980] LustreError: 46116:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1212.047678] LustreError: 46116:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1212.198539] Lustre: server umount lustre-MDT0001 complete [ 1221.101597] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1223.604286] Key type lgssc unregistered [ 1223.883718] LNet: 46648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.888413] LNetError: 46648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.897729] LNet: Removed LNI 192.168.201.116@tcp [ 1224.555948] Key type .llcrypt unregistered [ 1224.557718] Key type ._llcrypt unregistered [ 1246.823448] Key type ._llcrypt registered [ 1246.825153] Key type .llcrypt registered [ 1246.981641] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1248.605547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.644662] alg: No test for adler32 (adler32-zlib) [ 1249.740776] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 1249.999214] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1251.744216] Key type lgssc registered [ 1253.105321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.902081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.926558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.580800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.275967] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1271.782735] LustreError: 48045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.805333] LustreError: 48045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1274.169618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1274.487064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1277.499048] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1282.616131] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1286.040633] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1292.302246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1292.635154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.422051] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1302.330758] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1307.009766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1312.603277] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1316.183668] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1328.399456] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1622h40m21s [ 1330.288589] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1334.050188] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1335.150194] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1335.785522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1335.789131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.868191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.879865] Lustre: Skipped 5 previous similar messages [ 1339.059647] LustreError: 50610:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1339.179389] Lustre: server umount lustre-MDT0000 complete [ 1340.175174] LustreError: 48041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.950452] LustreError: 48046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.990963] LustreError: 48046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1343.068589] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1346.323443] LustreError: 50010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.171963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.354650] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.654940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.693363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1351.658770] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1354.519942] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1358.085216] Lustre: DEBUG MARKER: Wait for df (36028) ... [ 1360.094839] Lustre: DEBUG MARKER: done [ 1365.474785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1365.478717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.490103] Lustre: Skipped 2 previous similar messages [ 1365.511499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1372.130915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1372.148960] Lustre: Skipped 4 previous similar messages [ 1378.272216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1378.306984] LustreError: 51667:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1378.312903] LustreError: 51667:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1378.454338] Lustre: server umount lustre-OST0000 complete [ 1382.375335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.381633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.382572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.382577] Lustre: Skipped 2 previous similar messages [ 1382.389613] Lustre: Skipped 2 previous similar messages [ 1382.408175] LustreError: Skipped 1 previous similar message [ 1388.511133] LustreError: 51869:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1388.514218] LustreError: 51869:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1388.550682] LustreError: 50010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.585532] LustreError: 50010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1388.742352] Lustre: server umount lustre-MDT0000 complete [ 1393.893825] LustreError: 48026:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350889 with bad export cookie 16497330522858674531 [ 1393.909521] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1394.125111] LustreError: 52071:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1394.137553] LustreError: 52071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1394.435507] Lustre: server umount lustre-MDT0001 complete [ 1405.345107] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1408.036387] Key type lgssc unregistered [ 1408.368663] LNet: 52603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1408.388774] LNetError: 52603:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.390507] LNet: Removed LNI 192.168.201.116@tcp [ 1410.291121] Key type .llcrypt unregistered [ 1410.292924] Key type ._llcrypt unregistered [ 1442.311752] Key type ._llcrypt registered [ 1442.313921] Key type .llcrypt registered [ 1442.414987] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 1456.249706] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1457.273448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.308357] alg: No test for adler32 (adler32-zlib) [ 1458.305402] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 1458.524426] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1460.178557] Key type lgssc registered [ 1461.355946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1471.604687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1478.124912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.901083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1490.328473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1497.743216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1497.836370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1497.857975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1499.092566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1499.125149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1499.207438] Lustre: lustre-MDT0000: new disk, initializing [ 1499.317485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.349363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1503.472951] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1513.543971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1513.641893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1513.721345] Lustre: 56559:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1513.778691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1513.782102] Lustre: Skipped 1 previous similar message [ 1513.840295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1513.874995] Lustre: lustre-MDT0001: new disk, initializing [ 1513.978942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1514.048020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1514.053473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1517.902610] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1525.504202] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1530.670934] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1538.051270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1538.126542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1538.409743] Lustre: lustre-OST0000: new disk, initializing [ 1538.414669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1538.519598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.190469] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1548.339612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1548.353425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1548.466156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1551.628056] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1555.432595] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1555.605806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1559.233541] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1559.434216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1564.133096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1564.133210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.143212] LustreError: Skipped 1 previous similar message [ 1564.159827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.250598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.260340] Lustre: Skipped 2 previous similar messages [ 1574.373087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1574.382821] Lustre: Skipped 1 previous similar message [ 1575.392483] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1575.430265] LustreError: 59493:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1575.530543] Lustre: server umount lustre-OST0000 complete [ 1579.493201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.517397] Lustre: Skipped 1 previous similar message [ 1579.523634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.535760] Lustre: Skipped 1 previous similar message [ 1584.703556] LustreError: 59696:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1584.708020] LustreError: 59696:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1584.801280] Lustre: server umount lustre-MDT0000 complete [ 1587.877281] LustreError: 56550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351083 with bad export cookie 1630944017924683943 [ 1587.883720] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1587.900471] LustreError: 56550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1588.029778] LustreError: 59897:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1588.037037] LustreError: 59897:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1588.174263] Lustre: server umount lustre-MDT0001 complete [ 1596.167409] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 22:44:50 (1763351090) [ 1602.102170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.590790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.677247] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1607.659191] LustreError: 60569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.666915] LustreError: 60569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1611.747362] LustreError: 60569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.729248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1614.996491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1618.481619] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1623.717097] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1627.132252] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1632.924036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1633.372657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1638.167267] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1643.170071] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1653.329976] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1622h45m46s [ 1654.983365] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1661.793213] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1664.438352] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1664.566328] LustreError: 60563:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1684.803670] Lustre: lustre-MDT0000: Client c7d66a05-aea0-4d6e-8c88-5c0c7b2c09cd (at 192.168.201.16@tcp) reconnecting [ 1704.632145] LustreError: 60563:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1706.614518] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1708.912860] Lustre: DEBUG MARKER: done [ 1713.634985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.635035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1713.641836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1713.641845] Lustre: Skipped 2 previous similar messages [ 1713.648965] Lustre: Skipped 2 previous similar messages [ 1719.603097] LustreError: 63574:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1719.608562] LustreError: 63574:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1719.687327] Lustre: server umount lustre-OST0000 complete [ 1723.876675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.897075] Lustre: Skipped 1 previous similar message [ 1728.877625] LustreError: 63777:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1728.880911] LustreError: 63777:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1728.945648] Lustre: server umount lustre-MDT0000 complete [ 1730.033359] LustreError: 60568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.058133] LustreError: 60568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1732.238167] LustreError: 60550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351227 with bad export cookie 1630944017924684825 [ 1732.242420] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.249745] LustreError: 60550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1732.472927] Lustre: server umount lustre-MDT0001 complete [ 1741.584788] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1743.674884] Key type lgssc unregistered [ 1743.930300] LNet: 64511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.943044] LNetError: 64511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.952742] LNet: Removed LNI 192.168.201.116@tcp [ 1744.685294] Key type .llcrypt unregistered [ 1744.687643] Key type ._llcrypt unregistered [ 1779.744818] Key type ._llcrypt registered [ 1779.748108] Key type .llcrypt registered [ 1779.859087] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 1794.173633] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1795.364245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.479938] alg: No test for adler32 (adler32-zlib) [ 1796.535382] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 1796.829252] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1798.521092] Key type lgssc registered [ 1799.457097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1808.808093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1815.405206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1821.212440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1827.558897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1833.968627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1834.040332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1834.063757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.250496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1835.291356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1835.407882] Lustre: lustre-MDT0000: new disk, initializing [ 1835.535877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.560430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1839.175606] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1848.375497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1848.432713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1848.499660] Lustre: 68462:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1848.562663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1848.566787] Lustre: Skipped 1 previous similar message [ 1848.656205] Lustre: lustre-MDT0001: new disk, initializing [ 1848.744789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1848.778561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1848.792067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1851.981663] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1858.223488] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1861.877346] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1867.373651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1867.448515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1867.698975] Lustre: lustre-OST0000: new disk, initializing [ 1867.709076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1867.795593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.751232] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1874.524705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1874.540735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1874.671053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1880.162959] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1884.036111] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1884.240562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1887.513865] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1887.678932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1890.273578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.287695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.317701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1895.401524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1895.408122] Lustre: Skipped 2 previous similar messages [ 1895.582105] LustreError: 71400:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1895.734299] Lustre: server umount lustre-OST0000 complete [ 1900.003769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1900.014958] LustreError: Skipped 1 previous similar message [ 1900.019409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.029577] Lustre: Skipped 1 previous similar message [ 1900.035662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.549334] LustreError: 71602:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1905.563430] LustreError: 71602:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1905.668490] Lustre: server umount lustre-MDT0000 complete [ 1909.386304] LustreError: 68453:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351404 with bad export cookie 15078670892481455927 [ 1909.388424] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.406989] LustreError: 68453:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1909.561866] LustreError: 71803:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1909.575072] LustreError: 71803:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1909.827273] Lustre: server umount lustre-MDT0001 complete [ 1918.165823] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 22:50:11 (1763351411) [ 1919.803264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1921.535293] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 22:50:15 (1763351415) [ 1932.509961] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1936.368330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.688918] LustreError: 72852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.721173] LustreError: 72852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1936.770587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.579803] Lustre: Failing over lustre-MDT0000 [ 1937.582873] LustreError: 72880:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1937.596539] LustreError: 72880:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1937.604153] LustreError: 72880:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1937.754433] LustreError: 72931:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1937.768995] LustreError: 72931:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1937.928549] Lustre: server umount lustre-MDT0000 complete [ 1943.913380] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:50:37 (1763351437) [ 1950.298867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.620710] LustreError: 73500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.664262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.091938] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1955.815981] LustreError: 73500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.940812] LustreError: 73500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.453859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1961.695716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1965.455202] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1970.501648] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1974.147701] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1980.134354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1980.418797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.296337] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1990.951839] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1999.843952] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2005.985867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.992186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.015587] Lustre: Skipped 2 previous similar messages [ 2006.037194] Lustre: Skipped 3 previous similar messages [ 2006.867323] LustreError: 75843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2006.870087] LustreError: 75843:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2006.935133] Lustre: server umount lustre-MDT0000 complete [ 2010.093662] LustreError: 75844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351505 with bad export cookie 15078670892481457082 [ 2010.095832] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.105941] LustreError: 75844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2010.304433] Lustre: server umount lustre-MDT0001 complete [ 2012.842682] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2017.391537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2025.630244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.360213] Lustre: 66670:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351506/real 1763351506] req@00000000a59373eb x1849007844429056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763351522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2027.394041] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.411279] Lustre: Skipped 1 previous similar message [ 2031.584489] Lustre: 66673:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351510/real 1763351510] req@0000000014ff4efa x1849007844429184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763351526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2031.611874] Lustre: 66673:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2035.112468] LustreError: 75106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.285033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.400839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2036.263268] Lustre: 66671:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351515/real 1763351515] req@00000000cd59aad3 x1849007844429568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763351531 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2039.001116] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2041.441097] Lustre: 66671:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351520/real 1763351520] req@00000000a580f13b x1849007844429696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763351536 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2045.923685] LustreError: 75106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.931312] LustreError: 75106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2045.933084] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2046.087692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2049.717911] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2051.571245] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2055.154526] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2058.822396] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2065.799854] LustreError: 78307:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2065.804539] LustreError: 78307:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2065.906523] Lustre: server umount lustre-MDT0000 complete [ 2066.917284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.940990] Lustre: Skipped 2 previous similar messages [ 2066.949771] LustreError: 76903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.967302] LustreError: 76903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2069.387904] LustreError: 73479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351564 with bad export cookie 15078670892481458986 [ 2069.390033] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.394023] LustreError: 73479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2069.729350] Lustre: server umount lustre-MDT0001 complete [ 2075.110868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2084.380385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.416532] Lustre: 66673:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351567/real 1763351567] req@00000000634a0c26 x1849007844453376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763351583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2088.444306] Lustre: 66673:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2088.448188] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.614336] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd142334957613886 [ 2093.631520] Lustre: MGC192.168.201.116@tcp: Connection restored to 0@lo (at 0@lo) [ 2093.633829] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2093.661636] Lustre: MGS: Client 29a00d0d-4b51-4187-86e4-2e15a98ffb76 (at 0@lo) reconnecting [ 2094.176586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2098.885788] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2100.326399] Lustre: 66672:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351579/real 1763351579] req@0000000014ff4efa x1849007844454016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763351595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2100.364612] Lustre: 66672:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2103.781582] LustreError: 79364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.808847] LustreError: 79364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2106.870238] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2106.879734] Lustre: Skipped 1 previous similar message [ 2108.733864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2109.160188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2109.176038] Lustre: Skipped 2 previous similar messages [ 2114.132994] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2114.535994] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2119.313434] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2122.705053] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2129.893636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.895595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.896726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2129.906110] Lustre: Skipped 2 previous similar messages [ 2129.908544] Lustre: Skipped 1 previous similar message [ 2130.059850] LustreError: 80765:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2130.062887] LustreError: 80765:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2130.133856] Lustre: server umount lustre-MDT0000 complete [ 2133.725358] LustreError: 73480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351629 with bad export cookie 15078670892481460358 [ 2133.728400] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.733640] LustreError: 73480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2133.982676] Lustre: server umount lustre-MDT0001 complete [ 2138.967959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2142.085199] Lustre: DEBUG MARKER: files compared the same [ 2154.267630] Lustre: server umount lustre-OST0000 complete [ 2168.263655] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2170.352186] Key type lgssc unregistered [ 2170.576871] LNet: 82400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.583883] LNetError: 82400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.595789] LNet: Removed LNI 192.168.201.116@tcp [ 2171.192968] Key type .llcrypt unregistered [ 2171.200985] Key type ._llcrypt unregistered [ 2190.426450] Key type ._llcrypt registered [ 2190.431923] Key type .llcrypt registered [ 2190.529550] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2191.097752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2191.113131] alg: No test for adler32 (adler32-zlib) [ 2192.114796] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2192.356200] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2194.048216] Key type lgssc registered [ 2195.009134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2201.460444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2201.487546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.861943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.164218] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2208.228941] LustreError: 83778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.239252] LustreError: 83778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2213.232458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2213.344784] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2213.453967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2216.638863] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2221.096267] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2224.518904] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2230.066817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2230.369621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.408764] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2239.632995] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2247.795578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2258.402602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2258.403746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.404832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2258.420500] Lustre: Skipped 1 previous similar message [ 2262.261139] LustreError: 86044:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2262.371171] Lustre: server umount lustre-OST0000 complete [ 2268.642948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.650056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.653357] Lustre: Skipped 2 previous similar messages [ 2271.670441] LustreError: 86246:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2271.674768] LustreError: 86246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2271.746989] Lustre: server umount lustre-MDT0000 complete [ 2273.762030] LustreError: 83772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.777271] LustreError: 83772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2274.606128] LustreError: 83758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351770 with bad export cookie 10002503404496013908 [ 2274.610478] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.613164] LustreError: 83758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2274.751927] LustreError: 86447:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2274.762272] LustreError: 86447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2274.886635] Lustre: server umount lustre-MDT0001 complete [ 2284.552459] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2286.524535] Key type lgssc unregistered [ 2286.726841] LNet: 86980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.732066] LNetError: 86980:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2286.742696] LNet: Removed LNI 192.168.201.116@tcp [ 2287.299903] Key type .llcrypt unregistered [ 2287.306042] Key type ._llcrypt unregistered [ 2306.125239] Key type ._llcrypt registered [ 2306.127045] Key type .llcrypt registered [ 2306.201920] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2307.055513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2307.062483] alg: No test for adler32 (adler32-zlib) [ 2308.049584] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2308.245385] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2309.912219] Key type lgssc registered [ 2310.844513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2316.557360] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2316.575311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2332.128332] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2337.252397] LustreError: 88328:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.116@tcp: failed processing log, type 4: rc = -110 [ 2363.872200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.727984] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2371.397525] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2376.785171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.941227] Lustre: *** cfs_fail_loc=706, val=0*** [ 2377.189814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.251066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2379.507299] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2384.356397] LustreError: 89303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.365723] LustreError: 89303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2385.266221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2385.571382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2388.454532] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2392.356388] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2395.014767] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2397.666233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2397.670496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.685884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.251617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.253640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.486135] LustreError: 90655:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2402.591967] Lustre: server umount lustre-OST0000 complete [ 2406.369173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.374056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.383070] Lustre: Skipped 1 previous similar message [ 2406.390524] Lustre: Skipped 1 previous similar message [ 2411.486997] LustreError: 90857:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2411.490021] LustreError: 90857:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2411.496890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2411.508520] Lustre: Skipped 2 previous similar messages [ 2411.584384] Lustre: server umount lustre-MDT0000 complete [ 2413.874033] LustreError: 88335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351909 with bad export cookie 1052568716971711260 [ 2413.878094] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2413.882080] LustreError: 88335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2413.968064] LustreError: 91058:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2413.970245] LustreError: 91058:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2414.071843] Lustre: server umount lustre-MDT0001 complete [ 2420.544410] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2422.300647] Key type lgssc unregistered [ 2422.482439] LNet: 91591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.487244] LNetError: 91591:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2422.502278] LNet: Removed LNI 192.168.201.116@tcp [ 2423.021590] Key type .llcrypt unregistered [ 2423.024556] Key type ._llcrypt unregistered [ 2437.931937] Key type ._llcrypt registered [ 2437.933584] Key type .llcrypt registered [ 2437.994574] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2438.520604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.580590] alg: No test for adler32 (adler32-zlib) [ 2439.554434] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2439.706279] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2441.328251] Key type lgssc registered [ 2441.912832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2446.485382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2446.502612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.679497] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2454.478550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2454.851416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.631172] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2460.133538] LustreError: 93637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.149533] LustreError: 93637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2462.650293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2462.837654] LustreError: 93637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.845500] LustreError: 93637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2462.851366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.930424] LustreError: 93640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.942518] LustreError: 93640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2465.813724] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2470.371039] LustreError: 93640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.380756] LustreError: 93640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2470.615304] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2470.622856] Lustre: MGS: Client 2cb26bad-117d-4434-9b1a-7a5cf40e219b (at 0@lo) reconnecting [ 2470.808500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.850171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2472.951469] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2486.245094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2486.245195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.246423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.256434] Lustre: Skipped 1 previous similar message [ 2489.588315] LustreError: 95600:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2489.660480] Lustre: server umount lustre-OST0000 complete [ 2496.481762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.481953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2496.482763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.482767] Lustre: Skipped 1 previous similar message [ 2496.486259] Lustre: Skipped 1 previous similar message [ 2497.678304] LustreError: 95801:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2497.682549] LustreError: 95801:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2497.694344] Lustre: server umount lustre-MDT0000 complete [ 2499.795626] LustreError: 96002:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2499.798653] LustreError: 96002:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2499.916277] Lustre: server umount lustre-MDT0001 complete [ 2502.273344] LustreError: 96203:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2502.280892] LustreError: 96203:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2502.318828] Lustre: server umount MGS complete [ 2509.190047] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:00:03 (1763352003) [ 2526.035278] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 2530.725110] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2536.072324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2539.996685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2543.595041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2547.533722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.704819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.743073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.086616] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2560.731164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2560.767774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2560.885884] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2560.922096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2560.966836] Lustre: lustre-MDT0001: new disk, initializing [ 2561.022478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2561.040564] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2561.044049] Lustre: 100715:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2561.049648] LustreError: 100715:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2563.382765] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2570.291158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2570.332204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2570.476165] Lustre: lustre-OST0000: new disk, initializing [ 2570.480385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2570.540595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.112240] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2577.442962] Lustre: 101926:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000109abeac x1849008518631680/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763352088 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2579.031460] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2582.705709] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.712528] Lustre: MGS: Client d05fa094-7658-4fe2-aa97-84552b3070fd (at 0@lo) reconnecting [ 2582.739458] Lustre: 100001:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2582.756739] LustreError: 100001:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2582.787276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2582.840661] Lustre: lustre-MDT0000: new disk, initializing [ 2582.934846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.951110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2583.971312] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2583.976106] Lustre: 102439:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000006448ed99 x1849008518637312/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763352095 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2585.195350] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2591.719287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2591.723609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2593.060855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2593.065368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2593.131086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2616.293776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.296242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2616.299964] Lustre: Skipped 1 previous similar message [ 2616.302332] Lustre: Skipped 1 previous similar message [ 2621.415282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2621.419511] Lustre: Skipped 2 previous similar messages [ 2622.134137] LustreError: 102960:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2622.145140] LustreError: 102960:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2622.216658] Lustre: server umount lustre-OST0000 complete [ 2626.529458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2626.532212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.539546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2626.544549] Lustre: Skipped 1 previous similar message [ 2626.555763] Lustre: Skipped 1 previous similar message [ 2630.545209] LustreError: 103163:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2630.552023] LustreError: 103163:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2630.568571] Lustre: server umount lustre-MDT0000 complete [ 2631.649514] LustreError: 100685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.660566] LustreError: 100685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2639.215931] Lustre: server umount lustre-MDT0001 complete [ 2641.508476] Lustre: server umount MGS complete [ 2648.931659] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:02:22 (1763352142) [ 2660.017773] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2661.541704] Key type lgssc unregistered [ 2661.735896] LNet: 104901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2661.740103] LNetError: 104901:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.760452] LNet: Removed LNI 192.168.201.116@tcp [ 2663.310259] Key type .llcrypt unregistered [ 2663.313872] Key type ._llcrypt unregistered [ 2673.710500] Key type ._llcrypt registered [ 2673.712079] Key type .llcrypt registered [ 2673.791659] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2674.345652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.398042] alg: No test for adler32 (adler32-zlib) [ 2675.355483] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2675.516706] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2677.160173] Key type lgssc registered [ 2677.845299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2682.327686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2682.342819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.673910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.896332] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2690.897886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2691.074112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2692.900356] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2695.772181] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2697.721859] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2701.675468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2701.881455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.497524] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2708.565349] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2709.041252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2724.321512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2724.325563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.333863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2726.382235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2726.385977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2726.392183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.695419] LustreError: 108349:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2728.761107] Lustre: server umount lustre-OST0000 complete [ 2731.489845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.493856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2731.500176] Lustre: Skipped 1 previous similar message [ 2731.505267] Lustre: Skipped 2 previous similar messages [ 2736.611516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.619027] Lustre: Skipped 1 previous similar message [ 2741.730296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.312221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2745.374508] LustreError: 108552:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2745.380081] LustreError: 108552:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2745.436402] Lustre: server umount lustre-MDT0000 complete [ 2745.826619] LustreError: 106084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2745.837046] LustreError: 106084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2747.824522] LustreError: 106063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352243 with bad export cookie 12611828842331733133 [ 2747.826382] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2747.830241] LustreError: 106063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2747.953332] LustreError: 108757:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2747.962667] LustreError: 108757:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2748.084193] Lustre: server umount lustre-MDT0001 complete [ 2754.268376] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2755.691066] Key type lgssc unregistered [ 2755.884335] LNet: 109289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2755.892712] LNetError: 109289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2755.908216] LNet: Removed LNI 192.168.201.116@tcp [ 2756.356145] Key type .llcrypt unregistered [ 2756.357708] Key type ._llcrypt unregistered [ 2766.350791] Key type ._llcrypt registered [ 2766.351956] Key type .llcrypt registered [ 2766.429083] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2766.957722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2767.009136] alg: No test for adler32 (adler32-zlib) [ 2767.967699] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2768.112364] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2769.720287] Key type lgssc registered [ 2770.372290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2774.392152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2774.395879] LustreError: 110315:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2774.448288] LustreError: 110332:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2774.451924] LustreError: 110315:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4949 [ 2774.459130] LustreError: 110315:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2774.465673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.470220] LustreError: 110315:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2775.791378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2780.071501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2780.241591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2782.243479] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2784.571396] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing lsmod [ 2786.101163] LustreError: 111507:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2786.112741] LustreError: 111508:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2786.116729] LustreError: 111507:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 2786.120485] LustreError: 111507:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2786.126307] LustreError: 111507:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2786.129294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2786.274971] LustreError: 111524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.285724] LustreError: 111524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2786.345099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.119508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.122796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2790.126553] Lustre: Skipped 2 previous similar messages [ 2790.135345] Lustre: Skipped 2 previous similar messages [ 2794.204959] LustreError: 111696:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2794.246906] Lustre: server umount lustre-MDT0000 complete [ 2795.496447] LustreError: 110367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.504061] LustreError: 110367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2796.371415] LustreError: 110348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352291 with bad export cookie 273310566252613592 [ 2796.375480] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.376583] LustreError: 110348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2796.385831] Lustre: 110362:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2798.368436] LustreError: 110802:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000573afd5f x1849008862544512/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2798.395583] LustreError: 111898:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2798.399930] LustreError: 111898:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2798.466402] Lustre: server umount lustre-MDT0001 complete [ 2809.857889] LustreError: 112099:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2809.863015] LustreError: 112099:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2809.930923] Lustre: server umount lustre-OST0000 complete [ 2814.761057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.091984] LustreError: 112576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.143333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.387045] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2820.582180] LustreError: 112576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.423956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2822.648707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.857131] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2827.740824] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.880391] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2833.746319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2833.948430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.998833] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2840.109744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2840.601354] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2855.395973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2855.399448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.408071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.465296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2858.469397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.470269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.848435] LustreError: 114845:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2860.851348] LustreError: 114845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2860.947132] Lustre: server umount lustre-OST0000 complete [ 2863.585572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2863.589277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.590982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.594328] Lustre: Skipped 1 previous similar message [ 2863.597690] Lustre: Skipped 1 previous similar message [ 2868.705605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.709700] Lustre: Skipped 2 previous similar messages [ 2869.539575] LustreError: 115046:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2869.543197] LustreError: 115046:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2869.600994] Lustre: server umount lustre-MDT0000 complete [ 2871.759871] LustreError: 112556:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352367 with bad export cookie 273310566252614278 [ 2871.765438] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2871.768057] LustreError: 112556:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2871.933539] Lustre: server umount lustre-MDT0001 complete [ 2877.497719] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2879.132707] Key type lgssc unregistered [ 2879.307540] LNet: 115780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2879.315656] LNetError: 115780:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2879.328430] LNet: Removed LNI 192.168.201.116@tcp [ 2879.814297] Key type .llcrypt unregistered [ 2879.816985] Key type ._llcrypt unregistered [ 2893.034969] Key type ._llcrypt registered [ 2893.036340] Key type .llcrypt registered [ 2893.099784] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2893.593150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2893.640334] alg: No test for adler32 (adler32-zlib) [ 2894.569808] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2894.733623] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2896.353300] Key type lgssc registered [ 2896.987664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2901.041707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2901.054679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.283503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.326634] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2909.201665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2909.467623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.633138] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2914.853117] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2917.323596] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2921.382456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2921.616139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.791416] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2926.768812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2928.348187] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2935.755309] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2937.437701] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2942.433397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2942.436904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.443488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.483488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2944.485930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.487784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.600493] LustreError: 119508:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2945.688281] Lustre: server umount lustre-OST0000 complete [ 2949.602299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.604223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.612696] Lustre: Skipped 1 previous similar message [ 2949.616807] Lustre: Skipped 1 previous similar message [ 2954.723549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.727083] Lustre: Skipped 2 previous similar messages [ 2959.841477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2959.844214] Lustre: Skipped 1 previous similar message [ 2962.400189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2962.448859] LustreError: 119710:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2962.454424] LustreError: 119710:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2962.495419] Lustre: server umount lustre-MDT0000 complete [ 2964.691276] LustreError: 117137:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352460 with bad export cookie 6498463941782185635 [ 2964.694345] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.699777] LustreError: 117137:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2964.798289] LustreError: 119913:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2964.802343] LustreError: 119913:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2964.909829] Lustre: server umount lustre-MDT0001 complete [ 2971.241045] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2972.752049] Key type lgssc unregistered [ 2972.931740] LNet: 120444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2972.936983] LNetError: 120444:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2972.950805] LNet: Removed LNI 192.168.201.116@tcp [ 2973.384718] Key type .llcrypt unregistered [ 2973.386085] Key type ._llcrypt unregistered [ 2983.226739] Key type ._llcrypt registered [ 2983.228382] Key type .llcrypt registered [ 2983.295700] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2983.723380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2983.744076] alg: No test for adler32 (adler32-zlib) [ 2984.670465] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 2984.790624] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2986.400813] Key type lgssc registered [ 2986.938045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2991.589363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2991.600131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.899084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.976454] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2999.819575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3000.050920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3002.284950] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3005.281787] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3007.504249] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3011.338839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3011.536651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.472365] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3017.433068] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3017.715318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3033.061398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3033.067936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.078598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3035.110926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3035.113754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3035.121083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.139134] LustreError: 123896:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3038.218988] Lustre: server umount lustre-OST0000 complete [ 3041.254223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3041.260997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.270577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.274287] Lustre: Skipped 1 previous similar message [ 3045.354044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.354846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3046.753688] LustreError: 124099:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3046.757523] LustreError: 124099:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3046.829456] Lustre: server umount lustre-MDT0000 complete [ 3049.397551] LustreError: 121605:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352544 with bad export cookie 12547008335136365575 [ 3049.400344] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.412184] LustreError: 121605:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3049.518115] LustreError: 124300:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3049.520812] LustreError: 124300:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3049.634063] Lustre: server umount lustre-MDT0001 complete [ 3056.439537] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3058.319441] Key type lgssc unregistered [ 3058.503988] LNet: 124830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3058.508813] LNetError: 124830:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3058.522981] LNet: Removed LNI 192.168.201.116@tcp [ 3059.031085] Key type .llcrypt unregistered [ 3059.034659] Key type ._llcrypt unregistered [ 3072.594064] Key type ._llcrypt registered [ 3072.595569] Key type .llcrypt registered [ 3072.676404] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3073.239502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.281963] alg: No test for adler32 (adler32-zlib) [ 3074.179656] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 3074.287307] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3075.888205] Key type lgssc registered [ 3076.462615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3080.454126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3080.477581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.735136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.743538] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3088.635756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3088.834040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.856471] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3093.881139] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3095.883146] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3099.521622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3099.692833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.364691] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3105.150192] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3106.867735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3118.510554] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3120.367754] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3120.370219] Lustre: Skipped 2 previous similar messages [ 3122.972466] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3122.975981] Lustre: Skipped 2 previous similar messages [ 3127.321796] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3127.339602] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3130.616217] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3130.620616] Lustre: Skipped 2 previous similar messages [ 3137.033294] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.16@tcp 192.168.0.[2,10]@tcp [ 3140.312528] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.16/24@tcp in log lustre-MDT0000 [ 3140.316875] Lustre: Skipped 2 previous similar messages [ 3146.250648] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.16/24@tcp [ 3146.253375] Lustre: Skipped 1 previous similar message [ 3149.796636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3149.799138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.803521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3149.808541] Lustre: Skipped 1 previous similar message [ 3149.815039] Lustre: Skipped 1 previous similar message [ 3154.920910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3154.929094] Lustre: Skipped 1 previous similar message [ 3155.509107] LustreError: 130067:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3155.589170] Lustre: server umount lustre-OST0000 complete [ 3160.036373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.038981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3160.043911] Lustre: Skipped 1 previous similar message [ 3165.155592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.158248] Lustre: Skipped 2 previous similar messages [ 3170.274552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.277941] Lustre: Skipped 1 previous similar message [ 3172.320141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3172.370334] LustreError: 130268:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3172.374220] LustreError: 130268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3172.410992] Lustre: server umount lustre-MDT0000 complete [ 3174.515446] LustreError: 128524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352669 with bad export cookie 9883160028659953635 [ 3174.517872] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3174.521134] LustreError: 128524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3174.621904] LustreError: 130474:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3174.628403] LustreError: 130474:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3174.745342] Lustre: server umount lustre-MDT0001 complete [ 3180.868715] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3182.436175] Key type lgssc unregistered [ 3182.614735] LNet: 131008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3182.623782] LNetError: 131008:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3183.656975] LNet: Removed LNI 192.168.201.116@tcp [ 3184.120451] Key type .llcrypt unregistered [ 3184.123946] Key type ._llcrypt unregistered [ 3202.318713] Key type ._llcrypt registered [ 3202.320461] Key type .llcrypt registered [ 3202.390963] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3202.951866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3203.024606] alg: No test for adler32 (adler32-zlib) [ 3203.997838] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 3204.124677] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3205.752550] Key type lgssc registered [ 3206.429277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3211.190483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3211.209716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.472260] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3212.485372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.487882] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.16/24@tcp [ 3214.752093] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3220.003756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3220.193973] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3220.204808] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.16/24@tcp [ 3220.239715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3222.469366] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3225.724956] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3227.904336] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3231.675055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3231.869971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.758938] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3236.973567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3237.489777] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3252.706600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3252.710181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.717243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3255.778548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3255.783502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3255.791203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.799825] Lustre: Skipped 1 previous similar message [ 3257.784930] LustreError: 134849:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3257.855867] Lustre: server umount lustre-OST0000 complete [ 3260.898741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.907978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3260.910764] Lustre: Skipped 1 previous similar message [ 3266.018746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.024870] Lustre: Skipped 1 previous similar message [ 3266.317986] LustreError: 135050:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3266.323197] LustreError: 135050:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3266.364282] Lustre: server umount lustre-MDT0000 complete [ 3268.362235] LustreError: 132510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352763 with bad export cookie 8111719619694293280 [ 3268.363568] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3268.370462] LustreError: 132510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3268.450886] LustreError: 135252:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3268.453263] LustreError: 135252:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3268.535086] Lustre: server umount lustre-MDT0001 complete [ 3274.370110] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3275.780393] Key type lgssc unregistered [ 3275.952608] LNet: 135783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3275.956261] LNetError: 135783:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3275.969642] LNet: Removed LNI 192.168.201.116@tcp [ 3276.466557] Key type .llcrypt unregistered [ 3276.468269] Key type ._llcrypt unregistered [ 3291.340373] Key type ._llcrypt registered [ 3291.342755] Key type .llcrypt registered [ 3291.411255] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3299.798454] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3300.238917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.284687] alg: No test for adler32 (adler32-zlib) [ 3301.201449] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 3301.315749] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3302.920180] Key type lgssc registered [ 3303.483564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3307.870652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3311.140082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3314.183268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3317.371955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3320.726707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3320.754380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3320.764461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.921779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3321.950211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3321.991948] Lustre: lustre-MDT0000: new disk, initializing [ 3322.076420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3322.090642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3324.142709] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3330.333681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3330.376637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3330.432250] Lustre: 139133:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3330.470110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3330.472977] Lustre: Skipped 1 previous similar message [ 3330.520288] Lustre: lustre-MDT0001: new disk, initializing [ 3330.586376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3330.611797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3330.623703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3332.717473] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3337.498274] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.872670] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3344.220201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3344.254885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3344.415850] Lustre: lustre-OST0000: new disk, initializing [ 3344.418387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3344.492257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.237893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3345.244928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3345.312810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3347.753147] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3352.461581] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3354.580964] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3354.686139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3356.608893] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3356.713804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3360.738781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3360.739475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.742242] LustreError: Skipped 1 previous similar message [ 3360.742964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3360.747607] Lustre: Skipped 1 previous similar message [ 3365.861862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3365.866757] Lustre: Skipped 2 previous similar messages [ 3370.979442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.984765] Lustre: Skipped 1 previous similar message [ 3372.000261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3372.024671] LustreError: 142074:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3372.087245] Lustre: server umount lustre-OST0000 complete [ 3376.100386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.102213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3376.106151] Lustre: Skipped 1 previous similar message [ 3381.218424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3381.221416] Lustre: Skipped 1 previous similar message [ 3388.384144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3388.445143] LustreError: 142276:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3388.449294] LustreError: 142276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3388.499592] Lustre: server umount lustre-MDT0000 complete [ 3390.589285] LustreError: 139126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352886 with bad export cookie 7830671333600336110 [ 3390.591843] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3390.594594] LustreError: 139126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3390.677737] LustreError: 142480:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3390.681243] LustreError: 142480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3390.747079] Lustre: server umount lustre-MDT0001 complete [ 3394.694923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.940737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.806678] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3400.162206] LustreError: 142955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3400.174977] LustreError: 142955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3401.320678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3401.546123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3403.382926] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3405.829268] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3407.695423] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3411.499103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3411.732511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.321299] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3418.869925] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3420.965341] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3421.075110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3423.143041] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3426.356057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3432.374586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3432.424554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3432.523297] Lustre: lustre-OST0001: new disk, initializing [ 3432.526581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3432.573766] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3435.686929] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3439.147718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3439.150974] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3439.195139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3440.041769] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3440.164323] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3442.230738] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3442.360923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3444.234578] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0b6c4717800.ost_server_uuid 50 [ 3444.938992] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0b6c4717800.ost_server_uuid in FULL state after 0 sec [ 3449.313389] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.315973] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.317855] LustreError: Skipped 1 previous similar message [ 3449.319749] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3449.324102] Lustre: Skipped 1 previous similar message [ 3449.331686] Lustre: Skipped 3 previous similar messages [ 3454.077261] LustreError: 147222:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3454.080514] LustreError: 147222:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3454.115423] Lustre: server umount lustre-OST0001 complete [ 3454.438504] LustreError: 144553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3454.453220] LustreError: 144553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3456.993854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3456.997309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.559802] LustreError: 146576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.568649] LustreError: 146576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3462.391263] LustreError: 147423:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3462.401000] LustreError: 147423:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3462.453552] Lustre: server umount lustre-OST0000 complete [ 3464.673464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.680991] Lustre: Skipped 2 previous similar messages [ 3469.794900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.806145] Lustre: Skipped 6 previous similar messages [ 3470.610325] LustreError: 147625:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3470.613633] LustreError: 147625:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3470.677416] Lustre: server umount lustre-MDT0000 complete [ 3472.620942] LustreError: 142934:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352968 with bad export cookie 7830671333600336999 [ 3472.621762] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.627989] LustreError: 142934:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3472.795734] Lustre: server umount lustre-MDT0001 complete [ 3482.041420] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3483.339342] Key type lgssc unregistered [ 3483.492646] LNet: 148814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3483.496402] LNetError: 148814:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3483.507611] LNet: Removed LNI 192.168.201.116@tcp [ 3483.895793] Key type .llcrypt unregistered [ 3483.897169] Key type ._llcrypt unregistered [ 3499.031489] Key type ._llcrypt registered [ 3499.032879] Key type .llcrypt registered [ 3499.079514] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3507.961996] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3508.490454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3508.503678] alg: No test for adler32 (adler32-zlib) [ 3509.434196] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 3509.553564] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3511.168205] Key type lgssc registered [ 3511.689425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3516.130023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3519.088751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3521.947779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3524.820419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3527.843622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3527.866226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3527.874542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.998364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3529.016593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3529.060215] Lustre: lustre-MDT0000: new disk, initializing [ 3529.104448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.116806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3530.890718] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3535.787116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3535.819477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3535.866435] Lustre: 152738:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3535.896882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3535.899660] Lustre: Skipped 1 previous similar message [ 3535.940048] Lustre: lustre-MDT0001: new disk, initializing [ 3535.985269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3536.003293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3536.008951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3537.693049] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3541.290305] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3542.905294] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3545.735463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3545.768169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3545.904955] Lustre: lustre-OST0000: new disk, initializing [ 3545.907166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3545.943425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.696262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3546.699976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3546.758932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3548.350184] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3552.025985] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3553.902436] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3553.992383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3555.733389] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3555.829348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3557.348122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.348745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3557.359518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3557.362815] LustreError: Skipped 1 previous similar message [ 3557.368260] Lustre: Skipped 1 previous similar message [ 3562.468211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3562.476411] Lustre: Skipped 1 previous similar message [ 3562.739115] LustreError: 155673:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3562.817123] Lustre: server umount lustre-OST0000 complete [ 3567.075914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3567.079755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.085993] Lustre: Skipped 1 previous similar message [ 3567.090843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3570.642838] LustreError: 155874:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3570.644921] LustreError: 155874:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3570.715391] Lustre: server umount lustre-MDT0000 complete [ 3572.443246] LustreError: 152729:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353067 with bad export cookie 10782148389885435626 [ 3572.446155] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3572.449303] LustreError: 152729:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3572.536663] LustreError: 156076:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3572.540796] LustreError: 156076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3572.616470] Lustre: server umount lustre-MDT0001 complete [ 3577.317309] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 23:17:51 (1763353071) [ 3585.418145] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3589.860222] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3594.965456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3598.407694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3601.420691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3604.619233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3608.036108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3608.076473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.203516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3608.225810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3608.268827] Lustre: lustre-MDT0000: new disk, initializing [ 3608.315611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.329694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3610.250932] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3615.480340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3615.514459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3615.560881] Lustre: 159062:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3615.591697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3615.594440] Lustre: Skipped 1 previous similar message [ 3615.635737] Lustre: lustre-MDT0001: new disk, initializing [ 3615.683397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3615.702595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3615.707154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3617.488480] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3621.428752] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3623.070067] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3626.176681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3626.216290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3626.342713] Lustre: lustre-OST0000: new disk, initializing [ 3626.345892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3626.395977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3628.854454] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3630.338497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3630.342739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3630.387756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3632.770419] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3635.091611] Lustre: Failing over lustre-OST0000 [ 3635.123169] LustreError: 161504:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3635.126268] LustreError: 161504:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3635.174691] Lustre: server umount lustre-OST0000 complete [ 3649.279050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3649.452090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.457161] Lustre: Skipped 1 previous similar message [ 3649.469566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.482495] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3652.241785] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3652.576120] Lustre: 150956:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763353131/real 1763353131] req@000000006719c0b0 x1849009639580672/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763353147 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3652.585105] Lustre: 150956:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3654.024323] Lustre: Failing over lustre-MDT0000 [ 3654.103723] LustreError: 162477:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3654.106835] LustreError: 162477:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3654.153751] Lustre: server umount lustre-MDT0000 complete [ 3654.423386] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3654.627451] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.635672] Lustre: Skipped 3 previous similar messages [ 3656.672160] Lustre: 150954:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763353136/real 1763353136] req@00000000da8ce8ad x1849009639582464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763353152 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3659.524901] LustreError: 159067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.532796] LustreError: 159067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3661.792187] Lustre: 150955:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763353141/real 1763353141] req@0000000084354c3c x1849009639583616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763353157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3664.645123] LustreError: 159069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3664.651522] LustreError: 159069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3667.795686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.853312] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3667.858516] Lustre: 159064:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3667.944439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3668.001140] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3669.520119] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3669.766143] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3673.066628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3673.074638] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3673.101506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3678.181970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3678.185144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.185441] LustreError: Skipped 1 previous similar message [ 3678.188245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3678.188250] Lustre: Skipped 1 previous similar message [ 3678.194856] Lustre: Skipped 1 previous similar message [ 3680.565081] LustreError: 163359:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3680.567963] LustreError: 163359:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3680.634149] Lustre: server umount lustre-OST0000 complete [ 3683.298893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.299296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3683.301725] Lustre: Skipped 1 previous similar message [ 3688.699740] Lustre: server umount lustre-MDT0000 complete [ 3690.613977] LustreError: 159052:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353186 with bad export cookie 10782148389885437698 [ 3690.615382] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3690.621756] LustreError: 159052:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3690.825051] Lustre: server umount lustre-MDT0001 complete [ 3695.561736] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3696.875660] Key type lgssc unregistered [ 3697.028511] LNet: 164294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3697.033689] LNetError: 164294:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3697.045890] LNet: Removed LNI 192.168.201.116@tcp [ 3697.430395] Key type .llcrypt unregistered [ 3697.432177] Key type ._llcrypt unregistered [ 3720.196309] Key type ._llcrypt registered [ 3720.198133] Key type .llcrypt registered [ 3720.258423] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3720.692919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.702358] alg: No test for adler32 (adler32-zlib) [ 3721.643830] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 3721.765793] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3723.384253] Key type lgssc registered [ 3723.904687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3727.794450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3727.809249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3729.048354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3730.843942] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3734.802444] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3737.187095] LustreError: 165822:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3737.191096] LustreError: 165822:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3737.194614] LustreError: 165822:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3737.269476] LustreError: 166344:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3737.352211] Lustre: server umount lustre-MDT0000 complete [ 3744.014706] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3748.936620] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3753.579930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.503186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3759.210517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3762.047636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3765.069652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3765.101257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3765.224090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3765.241890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3765.288541] Lustre: lustre-MDT0000: new disk, initializing [ 3765.338900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3765.349063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3767.141717] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3772.470958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3772.504138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3772.558318] Lustre: 168983:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3772.605257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3772.607929] Lustre: Skipped 1 previous similar message [ 3772.655485] Lustre: lustre-MDT0001: new disk, initializing [ 3772.702801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3772.721413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3772.726868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3774.376185] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3778.183477] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3779.797940] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3782.911808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3782.958859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3783.082291] Lustre: lustre-OST0000: new disk, initializing [ 3783.085444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3783.135434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.745530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3783.750079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3783.808699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3785.535688] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3789.602419] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3791.767212] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3792.507172] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:21:27 (1763353287) [ 3900.897054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3900.900392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.907580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.920530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.920864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.923563] Lustre: Skipped 1 previous similar message [ 3901.931226] Lustre: Skipped 1 previous similar message [ 3904.781163] LustreError: 172490:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3904.783494] LustreError: 172490:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3904.840182] Lustre: server umount lustre-MDT0000 complete [ 3906.472385] LustreError: 168975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353401 with bad export cookie 6260081033729226329 [ 3906.473143] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3906.478706] LustreError: 168975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3907.040868] LustreError: 172371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3907.042396] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.046915] LustreError: 172371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3907.052418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3907.057750] Lustre: Skipped 1 previous similar message [ 3911.072495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3911.072494] LustreError: 172371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.574497] LustreError: 172692:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3912.577476] LustreError: 172692:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3912.633397] Lustre: server umount lustre-MDT0001 complete [ 3920.303544] LustreError: 172893:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3920.306948] LustreError: 172893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3920.361479] Lustre: server umount lustre-OST0000 complete [ 3924.558473] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3928.735239] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3933.326552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.261519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3939.171686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3942.338163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3945.562038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3945.598470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.721363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3945.742054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3945.783824] Lustre: lustre-MDT0000: new disk, initializing [ 3945.836399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.850116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3947.606750] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3952.569026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3952.599603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3952.640580] Lustre: 175231:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3952.670625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3952.673264] Lustre: Skipped 1 previous similar message [ 3952.715097] Lustre: lustre-MDT0001: new disk, initializing [ 3952.776770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3952.781846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3954.412108] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3958.293627] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3959.860293] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3962.821442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3962.854240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3962.963282] Lustre: lustre-OST0000: new disk, initializing [ 3962.966661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3963.003038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.006420] Lustre: Skipped 1 previous similar message [ 3963.185977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3963.190604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3963.236113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3965.506298] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3969.442508] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3971.196094] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3971.292358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3973.061447] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3973.151955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3978.722264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3978.724059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.725657] LustreError: Skipped 1 previous similar message [ 3978.731902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3978.736071] Lustre: Skipped 1 previous similar message [ 3980.016105] LustreError: 178169:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3980.019090] LustreError: 178169:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3980.072738] Lustre: server umount lustre-OST0000 complete [ 3983.843564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3983.845131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.853821] Lustre: Skipped 2 previous similar messages [ 3987.917987] Lustre: server umount lustre-MDT0000 complete [ 3988.961545] LustreError: 175237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.968556] LustreError: 175237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3990.277450] LustreError: 175223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353485 with bad export cookie 6260081033729385145 [ 3990.281070] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.286132] LustreError: 175223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3990.411019] LustreError: 178571:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3990.415520] LustreError: 178571:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3990.500983] Lustre: server umount lustre-MDT0001 complete [ 3994.905320] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:24:49 (1763353489) [ 4002.582905] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 4006.768684] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4011.081620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4013.918711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4016.794113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4019.677297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4022.761122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4022.793574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.905565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4022.924318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4022.968677] Lustre: lustre-MDT0000: new disk, initializing [ 4023.022887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.034225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4024.902348] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4030.332311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4030.369504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4030.438585] Lustre: 181556:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4030.595961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4032.631556] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4036.942551] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4038.441034] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4041.294690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4041.327693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4043.898430] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4047.665441] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4050.443741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4050.491456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4057.785245] LustreError: 184100:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4057.788083] LustreError: 184100:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4057.851906] Lustre: server umount lustre-OST0000 complete [ 4060.641953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.643901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4060.647272] Lustre: Skipped 1 previous similar message [ 4060.662087] Lustre: Skipped 3 previous similar messages [ 4069.899485] Lustre: server umount lustre-MDT0000 complete [ 4070.881614] LustreError: 181568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.889520] LustreError: 181568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4071.637814] LustreError: 183307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353567 with bad export cookie 6260081033729386013 [ 4071.640536] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4071.642818] LustreError: 183307:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4082.216296] Lustre: server umount lustre-MDT0001 complete [ 4085.698868] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:26:20 (1763353580) [ 4092.188962] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 4095.943530] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4100.111368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4103.065769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4106.047677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4109.132417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4112.240816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4112.272260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.387132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4112.403213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4112.406232] Lustre: Skipped 3 previous similar messages [ 4112.448272] Lustre: lustre-MDT0000: new disk, initializing [ 4112.450293] Lustre: Skipped 2 previous similar messages [ 4112.485044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.487547] Lustre: Skipped 2 previous similar messages [ 4112.496790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4112.499752] Lustre: Skipped 2 previous similar messages [ 4114.169433] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4119.696362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4119.729856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4119.786261] Lustre: 187491:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4119.932748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4121.743359] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4125.682481] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4127.327917] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4130.393575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4130.431421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4133.024154] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4136.777732] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4140.622539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4145.636472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4145.638311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.638907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4145.638922] Lustre: Skipped 2 previous similar messages [ 4145.652517] Lustre: Skipped 1 previous similar message [ 4147.447247] LustreError: 190037:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4147.451414] LustreError: 190037:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4147.525080] Lustre: server umount lustre-OST0000 complete [ 4150.758175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4150.761296] LustreError: Skipped 1 previous similar message [ 4151.595461] Lustre: server umount lustre-MDT0000 complete [ 4153.347253] LustreError: 187484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353648 with bad export cookie 6260081033729387084 [ 4153.348221] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.352195] LustreError: 187484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4153.506216] Lustre: server umount lustre-MDT0001 complete [ 4157.713764] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4158.923348] Key type lgssc unregistered [ 4159.096527] LNet: 190971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4159.100695] LNetError: 190971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4159.113556] LNet: Removed LNI 192.168.201.116@tcp [ 4159.460678] Key type .llcrypt unregistered [ 4159.462489] Key type ._llcrypt unregistered [ 4169.855700] Key type ._llcrypt registered [ 4169.857167] Key type .llcrypt registered [ 4169.907756] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4170.390469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4170.399958] alg: No test for adler32 (adler32-zlib) [ 4171.293205] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 4171.399315] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4173.008322] Key type lgssc registered [ 4173.478739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4176.855749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4176.869477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4178.142449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4179.925636] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4183.728190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4183.876492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4185.369757] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4187.579594] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4189.037128] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4191.813744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4191.962205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.315197] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4196.703351] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4203.245359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4219.873259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4219.876976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4219.882415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4223.969957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4223.972693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4223.972909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.982362] Lustre: Skipped 1 previous similar message [ 4225.775094] LustreError: 194613:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4225.828597] Lustre: server umount lustre-OST0000 complete [ 4229.090798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.097337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4229.100118] Lustre: Skipped 1 previous similar message [ 4233.681388] LustreError: 194814:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4233.683709] LustreError: 194814:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4233.730088] Lustre: server umount lustre-MDT0000 complete [ 4234.209366] LustreError: 192345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.217729] LustreError: 192345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4235.372494] LustreError: 192326:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353730 with bad export cookie 13094222774778251756 [ 4235.375767] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4235.378618] LustreError: 192326:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4235.455948] LustreError: 195016:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4235.459173] LustreError: 195016:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4235.527275] Lustre: server umount lustre-MDT0001 complete [ 4240.010178] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4241.339405] Key type lgssc unregistered [ 4241.495424] LNet: 195547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4241.499372] LNetError: 195547:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4241.509818] LNet: Removed LNI 192.168.201.116@tcp [ 4241.920591] Key type .llcrypt unregistered [ 4241.922786] Key type ._llcrypt unregistered [ 4253.590185] Key type ._llcrypt registered [ 4253.591855] Key type .llcrypt registered [ 4253.638989] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4254.011843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4254.074111] alg: No test for adler32 (adler32-zlib) [ 4255.009819] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 4255.123110] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4256.736192] Key type lgssc registered [ 4257.203430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4260.598488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4260.609918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.849923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4263.717516] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4267.606062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4267.806412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4269.695883] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4272.321348] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4274.107532] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4277.168089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4277.361320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.744324] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4282.058688] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4287.593392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4292.889309] Lustre: lustre-OST0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 4293.091721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.092288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.099195] Lustre: Skipped 1 previous similar message [ 4296.306094] LustreError: 199172:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4296.349639] Lustre: server umount lustre-OST0000 complete [ 4298.832247] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa0b6c76fe000.ost_server_uuid 50 [ 4299.684811] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4310.210830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.211407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4310.214953] Lustre: Skipped 1 previous similar message [ 4310.216873] Lustre: Skipped 2 previous similar messages [ 4313.313950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4313.316935] Lustre: Skipped 1 previous similar message [ 4316.353201] LustreError: 199520:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4316.356086] LustreError: 199520:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4316.393401] Lustre: server umount lustre-MDT0000 complete [ 4317.927252] LustreError: 198543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353813 with bad export cookie 16296431345940316444 [ 4317.930190] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4317.933342] LustreError: 198543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4318.006729] LustreError: 199722:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4318.009590] LustreError: 199722:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4318.083791] Lustre: server umount lustre-MDT0001 complete [ 4321.925129] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:30:16 (1763353816) [ 4324.726450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4324.903630] LustreError: 200392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4324.912039] LustreError: 200392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4324.931429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.419424] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4329.953552] LustreError: 200392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.104186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4330.272712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.910526] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4334.193642] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4335.699565] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4338.412093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4338.572743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.998950] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4343.284929] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4346.150730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4346.190886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4346.279376] Lustre: lustre-OST0001: new disk, initializing [ 4346.282797] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4348.820302] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4349.212045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4349.217467] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4349.260625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4350.248769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4352.515360] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4355.041226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4355.044832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.050840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.728223] LustreError: 203597:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4359.730385] LustreError: 203597:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4359.749541] Lustre: server umount lustre-OST0000 complete [ 4360.673514] LustreError: 201991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.684055] LustreError: 201991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4362.126843] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4362.219321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4363.015129] LustreError: 201991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.872919] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4363.956387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4368.134347] LustreError: 201991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.142042] LustreError: 201991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4375.009223] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4375.012433] LustreError: Skipped 1 previous similar message [ 4375.014278] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.019294] Lustre: Skipped 1 previous similar message [ 4375.021566] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4375.023985] Lustre: Skipped 3 previous similar messages [ 4380.341297] LustreError: 204301:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4380.345112] LustreError: 204301:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4380.400203] Lustre: server umount lustre-OST0001 complete [ 4386.274608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.275253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4386.280603] Lustre: Skipped 2 previous similar messages [ 4386.283331] Lustre: Skipped 2 previous similar messages [ 4388.265835] Lustre: server umount lustre-MDT0000 complete [ 4389.806747] LustreError: 200373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353885 with bad export cookie 16296431345940317515 [ 4389.809763] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.812028] LustreError: 200373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4389.884965] LustreError: 204704:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4389.886879] LustreError: 204704:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4389.959741] Lustre: server umount lustre-MDT0001 complete [ 4392.335027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4394.034180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4395.714627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4397.482908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4400.554241] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:31:35 (1763353895) [ 4403.425072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.542882] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4403.560128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4403.641199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4403.644337] Lustre: Skipped 1 previous similar message [ 4405.166896] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4408.649806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4408.698794] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4408.715745] Lustre: 206196:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4410.423860] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4412.673330] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4414.178978] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4417.147986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4417.241659] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4419.703945] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4420.696384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4422.141945] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4425.248148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4425.288658] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4428.078352] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4430.607220] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4432.362770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.364675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4432.375092] Lustre: Skipped 1 previous similar message [ 4446.177430] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4446.206501] LustreError: 209390:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4446.208820] LustreError: 209390:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4446.239060] Lustre: server umount lustre-OST0000 complete [ 4446.486385] LustreError: 207825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.499944] LustreError: 207825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4457.956177] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.956182] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4457.956192] LustreError: Skipped 1 previous similar message [ 4463.075135] LustreError: 207824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.080412] LustreError: 207824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4463.376422] Lustre: server umount lustre-OST0001 complete [ 4465.120969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4465.124586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4465.127203] Lustre: Skipped 11 previous similar messages [ 4471.009317] Lustre: server umount lustre-MDT0000 complete [ 4472.407461] LustreError: 206929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353967 with bad export cookie 16296431345940318901 [ 4472.412435] LustreError: 206929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4472.413610] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4472.540447] Lustre: server umount lustre-MDT0001 complete [ 4474.754947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4476.349717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4477.934018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4479.452650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4481.911211] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:32:56 (1763353976) [ 4487.918341] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 4491.503759] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4495.034190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4497.436946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4499.847022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4502.452451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4505.547668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4505.590831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.721193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4505.745487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4505.791914] Lustre: lustre-MDT0000: new disk, initializing [ 4505.850347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.853297] Lustre: Skipped 3 previous similar messages [ 4505.863279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4507.601445] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4512.738953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4512.787984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4512.837784] Lustre: 213814:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4512.870435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4512.873054] Lustre: Skipped 1 previous similar message [ 4512.916618] Lustre: lustre-MDT0001: new disk, initializing [ 4512.982882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4512.989125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4514.665752] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4518.378320] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4519.867542] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4522.861245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4522.914736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4523.051685] Lustre: lustre-OST0000: new disk, initializing [ 4523.054490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4525.134028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4525.140353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4525.209095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4525.688237] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4529.327138] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4530.977436] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4531.071565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4532.658264] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4532.759443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4535.783863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.784123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4535.788452] Lustre: Skipped 3 previous similar messages [ 4535.789494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4535.797396] LustreError: Skipped 1 previous similar message [ 4535.799291] Lustre: Skipped 3 previous similar messages [ 4539.637127] LustreError: 216750:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4539.646441] LustreError: 216750:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4539.709281] Lustre: server umount lustre-OST0000 complete [ 4547.496424] Lustre: server umount lustre-MDT0000 complete [ 4549.042784] LustreError: 213807:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354044 with bad export cookie 16296431345940320448 [ 4549.047619] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4549.047935] LustreError: 213807:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4549.222833] Lustre: server umount lustre-MDT0001 complete [ 4552.721536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4552.910444] LustreError: 217628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.511647] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4558.062954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4559.862741] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4562.034760] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4563.479312] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4566.107060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4568.488182] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4570.620088] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4572.255731] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4579.513162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4581.088470] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4581.185959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4583.906322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4583.910450] LustreError: Skipped 2 previous similar messages [ 4584.105973] Lustre: server umount lustre-OST0000 complete [ 4586.875765] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4586.970447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4588.704807] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4588.795361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4613.476609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4613.641672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.645276] Lustre: Skipped 5 previous similar messages [ 4615.939382] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4618.107799] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4618.788227] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4626.102309] LustreError: 222090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4626.105829] LustreError: 222090:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4626.159575] Lustre: server umount lustre-OST0000 complete [ 4629.473947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.482289] Lustre: Skipped 6 previous similar messages [ 4642.272169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4642.335120] Lustre: server umount lustre-MDT0000 complete [ 4643.787290] LustreError: 218344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354139 with bad export cookie 16296431345940321330 [ 4643.789765] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4643.792211] LustreError: 218344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4643.914766] Lustre: server umount lustre-MDT0001 complete [ 4647.266399] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:35:41 (1763354141) [ 4649.993268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4650.187899] LustreError: 223167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.868469] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4655.760303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4657.631268] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4659.967850] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4661.473933] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4664.278811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4666.695427] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4668.853762] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4670.451793] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4675.681314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4677.389185] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4677.483708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4680.287255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4680.317869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4680.382157] Lustre: lustre-OST0001: new disk, initializing [ 4680.384353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4682.649315] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4686.195344] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4686.867787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4686.872526] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4686.907408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4687.942116] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4688.040382] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4689.802977] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4689.891933] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4691.939615] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4691.943222] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4691.944084] LustreError: Skipped 1 previous similar message [ 4691.946591] Lustre: Skipped 13 previous similar messages [ 4696.782427] Lustre: server umount lustre-OST0001 complete [ 4699.421291] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4699.534319] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4701.139203] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4701.231102] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4725.411599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4727.826713] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4730.179684] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4730.852321] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4730.855504] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4730.862430] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4730.862485] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4730.870451] Lustre: Skipped 1 previous similar message [ 4733.636069] Lustre: server umount lustre-OST0001 complete [ 4752.430704] Lustre: server umount lustre-OST0000 complete [ 4760.324927] LustreError: 229482:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4760.327276] LustreError: 229482:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4760.357173] Lustre: server umount lustre-MDT0000 complete [ 4761.803365] LustreError: 223148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354257 with bad export cookie 16296431345940322261 [ 4761.806303] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4761.808228] LustreError: 223148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4768.017855] Lustre: server umount lustre-MDT0001 complete [ 4770.426835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4772.099054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4773.677702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4775.171635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.909527] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:37:52 (1763354272) [ 4780.630731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.759230] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4780.774645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4780.865637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4780.868392] Lustre: Skipped 5 previous similar messages [ 4782.483396] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4786.150439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4786.211397] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4786.228239] Lustre: 231178:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4786.270928] LustreError: 231189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.278182] LustreError: 231189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 4787.973327] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4790.242297] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4791.807699] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4794.800737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4794.898845] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4797.336284] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4799.616848] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4809.390469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4809.428204] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4811.971338] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4814.249256] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4817.045367] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4818.789256] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4818.895522] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4820.594310] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4820.685351] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4822.404531] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0b6c4717800.ost_server_uuid 50 [ 4823.136812] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0b6c4717800.ost_server_uuid in FULL state after 0 sec [ 4823.668797] Lustre: Permanently deactivating lustre-OST0001 [ 4823.679979] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4825.068808] Lustre: Permanently reactivating lustre-OST0001 [ 4831.209169] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.211715] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4831.214580] Lustre: Skipped 9 previous similar messages [ 4831.219300] LustreError: Skipped 5 previous similar messages [ 4832.846330] Lustre: server umount lustre-OST0001 complete [ 4840.686941] Lustre: server umount lustre-OST0000 complete [ 4848.691141] Lustre: server umount lustre-MDT0000 complete [ 4850.409287] LustreError: 235012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354345 with bad export cookie 16296431345940323535 [ 4850.412482] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4850.415554] LustreError: 235012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4850.614871] Lustre: server umount lustre-MDT0001 complete [ 4853.450700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4855.208569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.904808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4858.638623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4861.769798] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:39:16 (1763354356) [ 4862.219035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4865.506849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.669478] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4865.688687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4865.693934] Lustre: Skipped 5 previous similar messages [ 4867.615233] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4871.394273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4871.440839] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4871.455789] Lustre: 237320:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4873.246180] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4875.509185] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4877.023287] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4879.954687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4879.995134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4880.095875] Lustre: Permanently deactivating lustre-OST0000 [ 4880.102332] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4880.757814] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4880.761852] LustreError: 239056:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 4882.377484] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4884.511186] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4887.596878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4888.669989] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4890.092694] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4892.360306] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4893.124702] Lustre: Permanently reactivating lustre-OST0000 [ 4894.167027] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4894.170293] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4894.174504] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4894.195496] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4894.204392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4911.045581] Lustre: server umount lustre-OST0001 complete [ 4918.608749] Lustre: server umount lustre-OST0000 complete [ 4926.249426] Lustre: server umount lustre-MDT0000 complete [ 4927.863401] LustreError: 237311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354423 with bad export cookie 16296431345940325180 [ 4927.867424] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4927.869486] LustreError: 237311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4928.018712] Lustre: server umount lustre-MDT0001 complete [ 4932.210640] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4933.471484] Key type lgssc unregistered [ 4933.625497] LNet: 241730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4933.628811] LNetError: 241730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4933.638653] LNet: Removed LNI 192.168.201.116@tcp [ 4933.995505] Key type .llcrypt unregistered [ 4933.996997] Key type ._llcrypt unregistered [ 4943.761806] Key type ._llcrypt registered [ 4943.763234] Key type .llcrypt registered [ 4943.810851] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4944.262720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4944.299927] alg: No test for adler32 (adler32-zlib) [ 4945.174212] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 4945.280047] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4946.872177] Key type lgssc registered [ 4947.271351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4948.559647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4951.147979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4951.158243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.374481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.046591] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4957.627543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4957.656896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4957.713154] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4957.727568] Lustre: 243144:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4957.733400] Lustre: Permanently deactivating lustre-MDT0001 [ 4957.742484] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4957.887728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4958.453494] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4958.457375] LustreError: 243186:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4958.463801] LustreError: 243186:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4958.468026] LustreError: 243186:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 4959.591957] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4962.119761] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4963.673853] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4966.558711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4966.706368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4967.716253] LustreError: 244771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.718614] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4967.722765] LustreError: 244771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4968.937197] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4971.178721] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4973.029298] LustreError: 244924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.280663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4974.401199] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4976.777940] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4977.518199] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4977.519259] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4977.523567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4979.072889] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4980.790209] Lustre: Permanently reactivating lustre-MDT0001 [ 4980.797372] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4980.800126] Lustre: Skipped 2 previous similar messages [ 4988.178514] Lustre: lustre-MDT0001: Client 58bb5886-862a-4e6f-af65-aab9c32b52f7 (at 192.168.201.16@tcp) reconnecting [ 4990.525915] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4991.183174] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4993.147881] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4993.233646] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4994.755201] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4994.840896] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4995.839611] Lustre: Permanently deactivating lustre-MDT0001 [ 4995.847318] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4995.850794] Lustre: Skipped 2 previous similar messages [ 5003.278436] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5006.786885] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5006.791227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.801521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5008.356266] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.356974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5008.361665] Lustre: Skipped 2 previous similar messages [ 5008.366029] Lustre: Skipped 3 previous similar messages [ 5012.928877] LustreError: 247219:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5012.959230] Lustre: server umount lustre-MDT0000 complete [ 5013.473578] LustreError: 243155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.481061] LustreError: 243155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5014.424285] LustreError: 243136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354509 with bad export cookie 1059241745938165729 [ 5014.428228] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5014.429709] LustreError: 243136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5014.504715] LustreError: 247423:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5014.507431] LustreError: 247423:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5014.568964] Lustre: server umount lustre-MDT0001 complete [ 5022.132853] LustreError: 247623:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5022.135505] LustreError: 247623:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5022.153649] Lustre: server umount lustre-OST0000 complete [ 5026.470944] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:42:01 (1763354521) [ 5030.557988] LustreError: 248627:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5030.560850] LustreError: 248627:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5030.608925] Lustre: server umount lustre-OST0001 complete [ 5032.919576] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5036.506757] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5040.545828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5043.129787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5045.622312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5048.166122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5051.026599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5051.066093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.182216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5051.187069] Lustre: Skipped 2 previous similar messages [ 5051.200125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5051.242116] Lustre: lustre-MDT0000: new disk, initializing [ 5051.285273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.296992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5052.853849] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5057.753537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5057.782236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5057.821149] Lustre: 250809:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5057.853854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5057.856425] Lustre: Skipped 1 previous similar message [ 5057.896174] Lustre: lustre-MDT0001: new disk, initializing [ 5057.951467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5057.956384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5059.503098] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5063.198605] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5064.766583] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5067.726579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5067.767049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5067.884620] Lustre: lustre-OST0000: new disk, initializing [ 5067.887150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5067.925152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5067.927858] Lustre: Skipped 1 previous similar message [ 5070.227179] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5072.397679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5072.402525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5072.445507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5074.069061] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5075.047361] LustreError: 250814:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5080.290844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5080.333032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5080.412044] Lustre: lustre-OST0001: new disk, initializing [ 5080.415227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5083.178812] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5085.064403] LustreError: 250814:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5085.074547] LustreError: 250814:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5087.266816] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5087.768082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5087.774420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5087.825198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5088.034542] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5095.168188] LustreError: 250814:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5097.954548] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5097.958324] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.965587] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5097.968384] Lustre: Skipped 1 previous similar message [ 5102.138268] LustreError: 254306:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5102.144105] LustreError: 254306:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5102.169709] Lustre: server umount lustre-OST0001 complete [ 5103.082543] LustreError: 252541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.089060] LustreError: 252541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5108.196401] LustreError: 253036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.204664] LustreError: 253036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5118.437633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.438099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5118.439376] LustreError: 252542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.439387] LustreError: 252542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5118.443293] Lustre: Skipped 1 previous similar message [ 5118.446272] LustreError: Skipped 1 previous similar message [ 5118.447184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5118.462972] Lustre: Skipped 2 previous similar messages [ 5121.329318] LustreError: 254527:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5121.331864] LustreError: 254527:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5121.377271] Lustre: server umount lustre-OST0000 complete [ 5123.554594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.556418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.559962] Lustre: Skipped 1 previous similar message [ 5123.564314] Lustre: Skipped 1 previous similar message [ 5133.793928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5133.796084] Lustre: Skipped 3 previous similar messages [ 5137.376146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5137.421049] LustreError: 254728:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5137.423273] LustreError: 254728:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5137.476240] Lustre: server umount lustre-MDT0000 complete [ 5137.890115] LustreError: 250820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5137.896857] LustreError: 250820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5139.167623] LustreError: 250799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354634 with bad export cookie 1059241745938168536 [ 5139.173121] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.173466] LustreError: 250799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5145.383989] Lustre: server umount lustre-MDT0001 complete [ 5150.000081] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5151.559516] Key type lgssc unregistered [ 5151.710589] LNet: 255462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5151.714139] LNetError: 255462:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5151.721497] LNet: Removed LNI 192.168.201.116@tcp [ 5152.118715] Key type .llcrypt unregistered [ 5152.119955] Key type ._llcrypt unregistered [ 5166.436576] Key type ._llcrypt registered [ 5166.438501] Key type .llcrypt registered [ 5166.489759] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5173.922825] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5174.387507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5174.398136] alg: No test for adler32 (adler32-zlib) [ 5175.295305] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5175.393863] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5176.992165] Key type lgssc registered [ 5177.468738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5181.282069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5184.505177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5187.386284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5190.161854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5193.172357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5193.192057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5193.198677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.311685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5194.330271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5194.373767] Lustre: lustre-MDT0000: new disk, initializing [ 5194.401282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.409287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5195.958371] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5200.665121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5200.695990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5200.739580] Lustre: 259385:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5200.767179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5200.770253] Lustre: Skipped 1 previous similar message [ 5200.814566] Lustre: lustre-MDT0001: new disk, initializing [ 5200.870783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5200.888821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5200.892914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5202.517461] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5206.207871] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5207.803892] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5210.793029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5210.825859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5210.956168] Lustre: lustre-OST0000: new disk, initializing [ 5210.958782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5210.999385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5213.476156] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5216.773352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5216.778245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5216.823751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5217.059717] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5218.761405] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5218.851938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5220.536029] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5220.641161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5221.857942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5221.859174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.861335] LustreError: Skipped 1 previous similar message [ 5221.867572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5221.871499] Lustre: Skipped 1 previous similar message [ 5223.471103] LustreError: 262322:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5223.521653] Lustre: server umount lustre-OST0000 complete [ 5226.465799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5226.469292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.474378] Lustre: Skipped 1 previous similar message [ 5226.476932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5231.243248] LustreError: 262523:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5231.247954] LustreError: 262523:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5231.300411] Lustre: server umount lustre-MDT0000 complete [ 5232.937628] LustreError: 259376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354728 with bad export cookie 12042437148492943754 [ 5232.941154] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.942196] LustreError: 259376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5233.013001] LustreError: 262725:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5233.015332] LustreError: 262725:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5233.089496] Lustre: server umount lustre-MDT0001 complete [ 5236.840366] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 23:45:31 (1763354731) [ 5239.812293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.030114] LustreError: 263395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.038102] LustreError: 263395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5240.067277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.817402] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5245.409133] LustreError: 263395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.657191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5245.823231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5247.502743] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5249.686338] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5251.287244] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5254.242589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5254.436930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5256.901336] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5259.272581] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5310.949155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.949472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5310.954434] Lustre: Skipped 1 previous similar message [ 5310.958302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5310.961862] Lustre: Skipped 2 previous similar messages [ 5314.999778] LustreError: 265742:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5315.002144] LustreError: 265742:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5315.072147] Lustre: server umount lustre-OST0000 complete [ 5316.831965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5321.660595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5321.803956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5323.897366] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5326.014824] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5331.045233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.049694] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.049770] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5331.056956] Lustre: Skipped 1 previous similar message [ 5362.146317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5362.148847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5362.151055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5362.155302] Lustre: Skipped 1 previous similar message [ 5365.743469] LustreError: 267405:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5365.745953] LustreError: 267405:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5365.806640] Lustre: server umount lustre-OST0000 complete [ 5367.265736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.272438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.274861] Lustre: Skipped 2 previous similar messages [ 5373.344068] Lustre: server umount lustre-MDT0000 complete [ 5374.661359] LustreError: 263374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354870 with bad export cookie 12042437148492944622 [ 5374.665482] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.665885] LustreError: 263374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5374.736031] LustreError: 267808:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5374.738523] LustreError: 267808:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5374.805382] Lustre: server umount lustre-MDT0001 complete [ 5378.536751] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5379.628709] Key type lgssc unregistered [ 5379.782564] LNet: 268340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5379.785565] LNetError: 268340:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5379.799402] LNet: Removed LNI 192.168.201.116@tcp [ 5380.152598] Key type .llcrypt unregistered [ 5380.154160] Key type ._llcrypt unregistered [ 5389.932588] Key type ._llcrypt registered [ 5389.934076] Key type .llcrypt registered [ 5389.979565] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5390.476809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5390.486245] alg: No test for adler32 (adler32-zlib) [ 5391.361905] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5391.461603] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5393.056177] Key type lgssc registered [ 5393.465360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5396.593724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5396.602431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.790130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.210980] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5402.510095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5402.685076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5404.296118] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5406.540342] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5408.033204] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5410.939420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5411.104177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.437702] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5415.594238] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5424.427884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5438.945152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.948488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.953598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.972839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.973359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5439.978606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5439.980891] Lustre: Skipped 1 previous similar message [ 5441.777096] LustreError: 272752:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5441.826468] Lustre: server umount lustre-OST0000 complete [ 5443.553420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5443.556406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.561187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.409997] LustreError: 272954:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5449.412503] LustreError: 272954:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5449.445146] Lustre: server umount lustre-MDT0000 complete [ 5450.209110] LustreError: 269715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5450.215718] LustreError: 269715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5450.858456] LustreError: 269694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354946 with bad export cookie 13327775309823533318 [ 5450.861229] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5450.862831] LustreError: 269694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5450.942041] LustreError: 273167:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5450.945488] LustreError: 273167:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5451.014919] Lustre: server umount lustre-MDT0001 complete [ 5455.088419] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5456.264288] Key type lgssc unregistered [ 5456.409117] LNet: 273698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5456.412498] LNetError: 273698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5456.425494] LNet: Removed LNI 192.168.201.116@tcp [ 5456.778537] Key type .llcrypt unregistered [ 5456.779822] Key type ._llcrypt unregistered [ 5464.603162] Key type ._llcrypt registered [ 5464.604573] Key type .llcrypt registered [ 5464.647543] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5464.976301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5465.016749] alg: No test for adler32 (adler32-zlib) [ 5465.876339] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5465.963511] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5467.544180] Key type lgssc registered [ 5467.969902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5471.091860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5471.101363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.327711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5473.852491] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5477.355710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5477.514406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5479.064776] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5481.127780] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5482.506233] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5485.004679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5485.143707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.262346] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5489.264255] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5497.449958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5508.067065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.068080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5508.073158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5508.075499] LustreError: Skipped 1 previous similar message [ 5509.875578] LustreError: 277287:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5509.926631] Lustre: server umount lustre-OST0000 complete [ 5513.186688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5513.188251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5513.191740] Lustre: Skipped 2 previous similar messages [ 5513.194075] Lustre: Skipped 1 previous similar message [ 5517.571804] LustreError: 277489:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5517.574075] LustreError: 277489:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5517.600920] Lustre: server umount lustre-MDT0000 complete [ 5518.305088] LustreError: 274896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.311690] LustreError: 274896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5518.927358] LustreError: 274859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355014 with bad export cookie 4302432115741860554 [ 5518.931981] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5518.935895] LustreError: 274859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5519.002894] LustreError: 277692:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5519.006115] LustreError: 277692:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5519.065775] Lustre: server umount lustre-MDT0001 complete [ 5522.984670] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5524.044499] Key type lgssc unregistered [ 5524.177506] LNet: 278222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5524.180690] LNetError: 278222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5524.189440] LNet: Removed LNI 192.168.201.116@tcp [ 5524.501443] Key type .llcrypt unregistered [ 5524.503213] Key type ._llcrypt unregistered [ 5534.629551] Key type ._llcrypt registered [ 5534.630923] Key type .llcrypt registered [ 5534.670074] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5535.135121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5535.175030] alg: No test for adler32 (adler32-zlib) [ 5536.039929] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5536.128091] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5537.712171] Key type lgssc registered [ 5538.090813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5541.111795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5541.121281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.348728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.982374] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5547.551956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5547.713456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5549.249947] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5551.390150] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5552.888790] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5555.566477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5555.695172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.903018] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5559.949292] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5566.955051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5582.306658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5582.308680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.310166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.314316] Lustre: Skipped 1 previous similar message [ 5585.582266] LustreError: 282640:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5585.625533] Lustre: server umount lustre-OST0000 complete [ 5587.425828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.427762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5587.430969] Lustre: Skipped 1 previous similar message [ 5587.433470] Lustre: Skipped 1 previous similar message [ 5592.545495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5592.547585] Lustre: Skipped 2 previous similar messages [ 5597.665107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5597.668128] Lustre: Skipped 1 previous similar message [ 5601.248134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5601.282625] LustreError: 282841:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5601.284704] LustreError: 282841:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5601.314743] Lustre: server umount lustre-MDT0000 complete [ 5601.762169] LustreError: 279597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.768466] LustreError: 279597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5602.769603] LustreError: 279577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355098 with bad export cookie 13723407123827363878 [ 5602.774634] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5602.774764] LustreError: 279577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5602.849486] LustreError: 283043:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5602.851825] LustreError: 283043:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5602.907786] Lustre: server umount lustre-MDT0001 complete [ 5606.782511] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5608.062482] Key type lgssc unregistered [ 5608.206029] LNet: 283574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5608.209882] LNetError: 283574:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5608.218108] LNet: Removed LNI 192.168.201.116@tcp [ 5608.562646] Key type .llcrypt unregistered [ 5608.564107] Key type ._llcrypt unregistered [ 5616.154812] Key type ._llcrypt registered [ 5616.156190] Key type .llcrypt registered [ 5616.199829] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5616.638119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5616.645329] alg: No test for adler32 (adler32-zlib) [ 5617.508214] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5617.597393] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5619.184109] Key type lgssc registered [ 5619.587744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5622.323636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5622.332093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.524426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5624.945169] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5628.270041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5628.431561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5629.902745] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5631.889458] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5633.334339] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5635.859680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5635.988702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.037876] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5640.064655] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5649.322759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5654.497510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5654.498449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.499756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5654.505823] Lustre: Skipped 1 previous similar message [ 5659.617235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5659.619756] Lustre: Skipped 2 previous similar messages [ 5660.525105] LustreError: 287197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5660.566764] Lustre: server umount lustre-OST0000 complete [ 5664.224977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5664.228624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.233933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5668.035349] LustreError: 287400:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5668.037359] LustreError: 287400:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5668.075957] Lustre: server umount lustre-MDT0000 complete [ 5669.526317] LustreError: 284735:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355164 with bad export cookie 13252251944192418278 [ 5669.528479] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.530713] LustreError: 284735:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5669.601150] LustreError: 287602:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5669.603520] LustreError: 287602:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5669.669683] Lustre: server umount lustre-MDT0001 complete [ 5673.531907] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5674.642680] Key type lgssc unregistered [ 5674.784541] LNet: 288132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5674.787414] LNetError: 288132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5674.799431] LNet: Removed LNI 192.168.201.116@tcp [ 5675.131780] Key type .llcrypt unregistered [ 5675.133481] Key type ._llcrypt unregistered [ 5678.938578] Key type ._llcrypt registered [ 5678.939657] Key type .llcrypt registered [ 5678.988985] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5684.925778] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5691.018896] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5691.466813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5691.495200] alg: No test for adler32 (adler32-zlib) [ 5692.345098] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5692.435996] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5694.024235] Key type lgssc registered [ 5694.420777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5697.390711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5699.602228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5701.558585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5703.707527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5706.047711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5706.072813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5706.078714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.167718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5707.181370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5707.217326] Lustre: lustre-MDT0000: new disk, initializing [ 5707.242580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.250594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5708.662694] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5712.972086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5713.002376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5713.045057] Lustre: 291615:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5713.077962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5713.080532] Lustre: Skipped 1 previous similar message [ 5713.123200] Lustre: lustre-MDT0001: new disk, initializing [ 5713.176413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5713.198506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5713.201859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5714.553195] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5717.832335] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5719.154824] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5721.463914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5721.488890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5721.616327] Lustre: lustre-OST0000: new disk, initializing [ 5721.619785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5721.628385] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5721.667639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5723.705663] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5726.713201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5726.716896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5726.729285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5726.800398] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5728.160163] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5728.237621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5729.584770] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5730.679253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 5731.809045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5731.811646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.815643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5735.396574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5735.397811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5735.399051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.390511] LustreError: 294560:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 5737.426074] Lustre: server umount lustre-OST0000 complete [ 5738.977117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5738.979474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5738.983276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5738.984953] Lustre: Skipped 1 previous similar message [ 5744.775403] LustreError: 294761:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5744.778296] LustreError: 294761:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5744.808464] Lustre: server umount lustre-MDT0000 complete [ 5746.062450] LustreError: 291607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355241 with bad export cookie 17453780904895725354 [ 5746.065302] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5746.066910] LustreError: 291607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5746.137361] LustreError: 294962:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5746.138977] LustreError: 294962:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5746.199109] Lustre: server umount lustre-MDT0001 complete [ 5749.020320] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:54:03 (1763355243) [ 5751.320576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.493471] LustreError: 295635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.502073] LustreError: 295635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5751.519054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.944450] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5755.998554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5756.153398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5757.626572] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5759.532478] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5760.780674] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5763.200268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5765.478396] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5767.492642] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5781.986400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5781.989340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.993133] Lustre: Skipped 1 previous similar message [ 5781.994954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5781.996428] Lustre: Skipped 2 previous similar messages [ 5786.085111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5786.086770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.117113] LustreError: 297902:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5787.119809] LustreError: 297902:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5787.157980] Lustre: server umount lustre-OST0000 complete [ 5791.202941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5791.205320] Lustre: Skipped 3 previous similar messages [ 5794.685511] LustreError: 298104:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5794.687871] LustreError: 298104:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5794.714604] Lustre: server umount lustre-MDT0000 complete [ 5796.056466] LustreError: 297251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355291 with bad export cookie 17453780904895726208 [ 5796.059468] LustreError: 296356:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5796.059968] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5796.061775] LustreError: 297251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5796.191750] Lustre: server umount lustre-MDT0001 complete [ 5799.926650] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5801.004852] Key type lgssc unregistered [ 5801.137531] LNet: 298839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5801.140597] LNetError: 298839:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5801.148461] LNet: Removed LNI 192.168.201.116@tcp [ 5801.459544] Key type .llcrypt unregistered [ 5801.461109] Key type ._llcrypt unregistered [ 5810.409340] Key type ._llcrypt registered [ 5810.410665] Key type .llcrypt registered [ 5810.452375] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5816.780954] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5817.163705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5817.170299] alg: No test for adler32 (adler32-zlib) [ 5818.038926] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 5818.135751] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5819.720968] Key type lgssc registered [ 5820.137423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5823.287653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5825.428473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5827.417047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.527110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5831.221876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5833.026910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5835.332742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5835.355661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5835.363199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5836.454245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5836.466533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5836.505413] Lustre: lustre-MDT0000: new disk, initializing [ 5836.530921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.538250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5837.854177] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5841.939384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5841.964530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5842.000243] Lustre: 302616:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5842.021043] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5842.023039] Lustre: Skipped 1 previous similar message [ 5842.059137] Lustre: lustre-MDT0001: new disk, initializing [ 5842.089759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5842.100511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5842.105135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5843.347513] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5846.387870] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5847.580267] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5849.850518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5849.877645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5849.990357] Lustre: lustre-OST2710: new disk, initializing [ 5849.992983] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5850.026600] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5850.620284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5850.623905] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5850.666447] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5851.891769] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5854.936951] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5857.156077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5857.178515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5857.227893] Lustre: lustre-OST03e8: new disk, initializing [ 5857.229587] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5857.260448] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5859.107726] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5860.858576] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5863.475271] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5863.552410] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5863.941653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5863.945965] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5863.979339] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5864.885251] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5864.961495] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5866.311735] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5866.396565] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5867.835274] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5867.917604] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5869.033093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.035817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.037028] Lustre: Skipped 2 previous similar messages [ 5869.040180] Lustre: Skipped 1 previous similar message [ 5873.120928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5873.123944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.129705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5874.146712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5874.150270] Lustre: Skipped 3 previous similar messages [ 5875.136757] LustreError: 306956:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5875.170145] Lustre: server umount lustre-MDT0000 complete [ 5876.418345] LustreError: 302606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355371 with bad export cookie 1831017922914186097 [ 5876.421776] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5876.422667] LustreError: 302606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5876.487289] LustreError: 307158:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5876.489655] LustreError: 307158:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5876.548879] Lustre: server umount lustre-MDT0001 complete [ 5888.500816] LustreError: 307359:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5888.502834] LustreError: 307359:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5888.522603] Lustre: server umount lustre-OST2710 complete [ 5899.953413] LustreError: 307561:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5899.955367] LustreError: 307561:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5899.998141] Lustre: server umount lustre-OST03e8 complete [ 5905.624034] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5908.498941] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5911.460740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5913.488662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5915.367537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5917.437246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5919.651946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5919.673862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.771964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5919.784334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5919.827267] Lustre: lustre-MDT0000: new disk, initializing [ 5919.851586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5919.859343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5921.220063] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5925.299269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5925.321037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5925.358028] Lustre: 310353:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5925.452985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5926.755905] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5929.888631] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5931.115072] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5933.380296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5933.404773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5933.503830] Lustre: lustre-OST0000: new disk, initializing [ 5933.505735] Lustre: Skipped 1 previous similar message [ 5933.507770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5933.510695] Lustre: Skipped 2 previous similar messages [ 5933.541391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.544035] Lustre: Skipped 1 previous similar message [ 5935.467706] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5938.508493] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5938.686018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5938.688382] Lustre: Skipped 1 previous similar message [ 5938.690276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5938.728931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5939.818704] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5939.889685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5941.223360] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5941.293252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5943.777959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5943.779714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.780969] LustreError: Skipped 1 previous similar message [ 5943.781534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5943.785758] Lustre: Skipped 1 previous similar message [ 5943.791389] Lustre: Skipped 1 previous similar message [ 5948.015040] LustreError: 313291:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5948.017149] LustreError: 313291:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5948.057345] Lustre: server umount lustre-OST0000 complete [ 5950.944813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5950.947482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.953447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5955.481754] Lustre: server umount lustre-MDT0000 complete [ 5956.714311] LustreError: 310345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355452 with bad export cookie 1831017922914187742 [ 5956.717177] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5956.719300] LustreError: 310345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5956.785644] LustreError: 313693:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5956.787731] LustreError: 313693:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5956.840712] Lustre: server umount lustre-MDT0001 complete [ 5959.472686] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:57:34 (1763355454) [ 5959.915803] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5960.482780] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:57:35 (1763355455) [ 5961.499077] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5964.185992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5965.439164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5966.704509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5967.963558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5968.320514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5970.612806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.721668] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5970.731225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5970.799388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5972.124204] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5974.917260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5974.951350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5974.960677] Lustre: 315817:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5976.409673] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5978.146140] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5979.246962] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5981.507661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5981.535109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5981.607295] Lustre: Denying initial registration attempt from nid 192.168.201.116@tcp, specified as failover [ 5981.611172] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5981.614168] LustreError: 317433:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 5981.616631] LustreError: 317433:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5981.618658] LustreError: 317433:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5981.645863] Lustre: server umount lustre-OST0000 complete [ 5981.647692] LustreError: 317433:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5985.248914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5985.250984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.254711] Lustre: Skipped 1 previous similar message [ 5985.256195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.258047] Lustre: Skipped 2 previous similar messages [ 5998.048138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5998.082743] LustreError: 317800:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5998.085358] LustreError: 317800:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5998.108043] Lustre: server umount lustre-MDT0000 complete [ 5999.326150] LustreError: 316543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355494 with bad export cookie 1831017922914188610 [ 5999.329748] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5999.330534] LustreError: 316543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5999.444856] Lustre: server umount lustre-MDT0001 complete [ 6002.863438] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6003.874873] Key type lgssc unregistered [ 6003.996322] LNet: 318536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6003.998706] LNetError: 318536:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6004.006377] LNet: Removed LNI 192.168.201.116@tcp [ 6004.319704] Key type .llcrypt unregistered [ 6004.321146] Key type ._llcrypt unregistered [ 6007.769181] Key type ._llcrypt registered [ 6007.770175] Key type .llcrypt registered [ 6007.805244] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6008.138141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6008.172874] alg: No test for adler32 (adler32-zlib) [ 6009.032957] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6009.116429] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6010.704159] Key type lgssc registered [ 6011.072095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6013.379432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6014.625910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6015.865097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6017.114048] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6017.475562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.825065] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6026.812609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6026.817826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.912480] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6027.919855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6027.982525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.237327] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6032.171623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6032.203137] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6032.215440] Lustre: 321266:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6032.319212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6033.653598] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6035.403866] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6036.644751] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6038.979629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6039.006983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6039.084672] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6039.155915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.136326] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6042.896157] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6046.690303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6046.692414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.692931] LustreError: Skipped 1 previous similar message [ 6046.698209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.701315] Lustre: Skipped 1 previous similar message [ 6049.648108] LustreError: 323550:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6049.687784] Lustre: server umount lustre-OST0000 complete [ 6051.809505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.810346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6051.813381] Lustre: Skipped 2 previous similar messages [ 6056.929577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6056.931832] Lustre: Skipped 2 previous similar messages [ 6057.086542] LustreError: 323751:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6057.089063] LustreError: 323751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6057.115921] Lustre: server umount lustre-MDT0000 complete [ 6058.374260] LustreError: 321258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355553 with bad export cookie 5710301169194681870 [ 6058.377527] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6058.377925] LustreError: 321258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6058.442359] LustreError: 323952:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6058.444346] LustreError: 323952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6058.501225] Lustre: server umount lustre-MDT0001 complete [ 6061.985798] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6063.044837] Key type lgssc unregistered [ 6063.180683] LNet: 324482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6063.183847] LNetError: 324482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6063.191400] LNet: Removed LNI 192.168.201.116@tcp [ 6063.508756] Key type .llcrypt unregistered [ 6063.510454] Key type ._llcrypt unregistered [ 6072.635309] Key type ._llcrypt registered [ 6072.636396] Key type .llcrypt registered [ 6072.673416] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6073.018398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6073.072679] alg: No test for adler32 (adler32-zlib) [ 6073.926051] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6074.012827] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6075.592190] Key type lgssc registered [ 6076.001052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6078.708343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6078.715323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.893319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.361204] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6084.204716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6084.351752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6085.825263] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6087.687839] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6088.878741] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6091.298590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6091.443735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.493665] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6095.440317] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6110.177362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6110.180470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.186087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6111.201164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6111.202485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.203973] Lustre: Skipped 1 previous similar message [ 6111.210457] Lustre: Skipped 1 previous similar message [ 6115.262631] LustreError: 328127:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6115.292555] Lustre: server umount lustre-MDT0000 complete [ 6116.322882] LustreError: 328026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6116.327830] LustreError: 328026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6116.588249] LustreError: 327477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355612 with bad export cookie 4451855911901841139 [ 6116.591973] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6116.592858] LustreError: 327477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6117.125647] LustreError: 325855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.133225] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 6117.135804] Lustre: Skipped 2 previous similar messages [ 6121.440920] LustreError: 325855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6121.441149] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.452035] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6122.686432] LustreError: 328329:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6122.688448] LustreError: 328329:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6122.746664] Lustre: server umount lustre-MDT0001 complete [ 6125.084112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6128.174877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.880766] LustreError: 325166:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004204dc72 x1849012329144064/t0(0) o250->MGC192.168.201.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6143.002913] LustreError: 327466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.010341] LustreError: 327466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6143.028632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.054138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6144.360305] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6147.542366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6147.698574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6149.173307] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6150.760718] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6151.061305] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6152.616356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 6153.847046] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6154.398359] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6156.258502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.260207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.262567] Lustre: Skipped 1 previous similar message [ 6156.264244] Lustre: Skipped 1 previous similar message [ 6166.498896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6166.501546] Lustre: Skipped 4 previous similar messages [ 6170.080145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6170.105414] LustreError: 330632:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6170.109564] LustreError: 330632:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6170.152808] Lustre: server umount lustre-OST0000 complete [ 6171.617494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.622387] Lustre: Skipped 1 previous similar message [ 6177.596635] LustreError: 330849:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6177.598222] LustreError: 330849:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6177.620262] Lustre: server umount lustre-MDT0000 complete [ 6178.920349] LustreError: 325839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355674 with bad export cookie 4451855911901848293 [ 6178.924491] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.925146] LustreError: 325839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6179.042165] Lustre: server umount lustre-MDT0001 complete [ 6182.743447] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6183.815958] Key type lgssc unregistered [ 6183.941410] LNet: 331581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6183.945155] LNetError: 331581:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6183.950322] LNet: Removed LNI 192.168.201.116@tcp [ 6184.272624] Key type .llcrypt unregistered [ 6184.274135] Key type ._llcrypt unregistered [ 6193.376057] Key type ._llcrypt registered [ 6193.377367] Key type .llcrypt registered [ 6193.418915] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6193.923614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6193.956419] alg: No test for adler32 (adler32-zlib) [ 6194.822897] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6194.906161] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6196.496207] Key type lgssc registered [ 6196.904846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6199.713114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6199.720847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.921542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6202.398332] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6205.401080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6205.530974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6206.958492] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6208.955608] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6210.180818] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6215.394317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.394948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6215.398344] Lustre: Skipped 1 previous similar message [ 6217.407218] LustreError: 334348:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6217.433634] Lustre: server umount lustre-MDT0000 complete [ 6218.738205] LustreError: 332939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355714 with bad export cookie 14472712091791278151 [ 6218.741691] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6218.741777] LustreError: 332939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6218.808548] LustreError: 334551:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6218.810723] LustreError: 334551:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6218.868476] Lustre: server umount lustre-MDT0001 complete [ 6223.948762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.050054] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6224.062263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6224.135740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6225.383779] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6228.379382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6228.416096] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6228.426376] Lustre: 335321:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6228.531748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6229.908475] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6231.774692] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6232.978389] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6235.583747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6235.658937] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6235.662788] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6235.667611] LustreError: 336978:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 6235.670561] LustreError: 336978:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6235.673537] LustreError: 336978:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6235.703763] Lustre: server umount lustre-OST0000 complete [ 6235.704911] LustreError: 336978:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6238.707976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6238.782936] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6238.858792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.815116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 6241.002903] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6243.110040] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6245.782901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6245.812514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6245.835586] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6245.877251] Lustre: lustre-OST0001: new disk, initializing [ 6245.879818] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6247.820129] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6249.280693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6249.284034] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6249.323711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6249.629933] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6254.562923] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.563386] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.565746] LustreError: Skipped 1 previous similar message [ 6254.569487] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6254.570782] Lustre: Skipped 1 previous similar message [ 6254.572209] Lustre: Skipped 1 previous similar message [ 6256.622107] LustreError: 339096:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6256.624787] LustreError: 339096:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6256.642270] Lustre: server umount lustre-OST0001 complete [ 6259.683791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.685142] LustreError: 337445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6259.685241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6259.685250] Lustre: Skipped 1 previous similar message [ 6259.688132] Lustre: Skipped 1 previous similar message [ 6259.693283] LustreError: 337445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6264.048797] LustreError: 339297:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6264.050425] LustreError: 339297:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6264.090621] Lustre: server umount lustre-OST0000 complete [ 6269.408857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6269.410842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.414135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6269.416811] Lustre: Skipped 1 previous similar message [ 6271.495015] LustreError: 339498:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6271.496638] LustreError: 339498:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6271.527257] Lustre: server umount lustre-MDT0000 complete [ 6272.816118] LustreError: 335311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355768 with bad export cookie 14472712091791278690 [ 6272.818758] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6272.819400] LustreError: 335311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6272.953655] Lustre: server umount lustre-MDT0001 complete [ 6276.432185] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6277.499292] Key type lgssc unregistered [ 6277.627435] LNet: 340231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6277.630662] LNetError: 340231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6277.639341] LNet: Removed LNI 192.168.201.116@tcp [ 6277.940165] Key type .llcrypt unregistered [ 6277.941548] Key type ._llcrypt unregistered [ 6288.855065] Key type ._llcrypt registered [ 6288.856569] Key type .llcrypt registered [ 6288.897246] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6294.700487] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6295.133916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6295.151231] alg: No test for adler32 (adler32-zlib) [ 6296.007025] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6296.100258] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6297.680162] Key type lgssc registered [ 6298.074232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6300.981538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6303.083783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6305.117715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6307.190384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6309.442695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6309.462988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6309.467886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6310.562601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6310.575246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6310.611969] Lustre: lustre-MDT0000: new disk, initializing [ 6310.635961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6310.642534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6311.952039] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6316.091754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6316.114411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6316.142344] Lustre: 344152:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6316.164682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6316.167021] Lustre: Skipped 1 previous similar message [ 6316.202534] Lustre: lustre-MDT0001: new disk, initializing [ 6316.238090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6316.251095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6316.255954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6317.545082] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6320.562856] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6321.749394] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6324.016074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6324.037274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6324.135354] Lustre: lustre-OST0000: new disk, initializing [ 6324.138178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6324.166171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.017657] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6329.076420] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6330.477115] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6332.411201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6332.413942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6332.445936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6332.596894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6333.971151] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6334.047738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6337.507557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6337.508588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.509380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.516320] Lustre: Skipped 1 previous similar message [ 6340.784103] LustreError: 347106:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6340.825407] Lustre: server umount lustre-OST0000 complete [ 6342.625056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.628638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.630242] Lustre: Skipped 1 previous similar message [ 6347.233059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6347.235745] LustreError: Skipped 1 previous similar message [ 6347.236934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.241481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6348.222465] LustreError: 347307:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6348.225141] LustreError: 347307:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6348.249321] Lustre: server umount lustre-MDT0000 complete [ 6349.512927] LustreError: 344143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355844 with bad export cookie 13680301601475707508 [ 6349.516440] LustreError: 344143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6349.517810] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6349.582927] LustreError: 347510:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6349.584828] LustreError: 347510:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6349.634874] Lustre: server umount lustre-MDT0001 complete [ 6352.457196] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 00:04:07 (1763355847) [ 6354.365441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6356.332972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.985093] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6366.865450] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6370.018831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.154195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.150355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.299707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.557036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.577728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.679084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6378.692971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6378.733662] Lustre: lustre-MDT0000: new disk, initializing [ 6378.761749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6378.770800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6380.096840] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6384.286678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.311673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6384.345938] Lustre: 351278:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6384.367809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6384.369587] Lustre: Skipped 1 previous similar message [ 6384.404585] Lustre: lustre-MDT0001: new disk, initializing [ 6384.437069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6384.449203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6384.452992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6385.760298] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6388.877712] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6390.121970] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6392.466740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.496835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6392.597140] Lustre: lustre-OST0000: new disk, initializing [ 6392.598923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6392.625975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6394.541952] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6397.658196] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6397.951855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6397.955827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6397.993207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6399.045598] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6399.117419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6400.483564] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6400.558314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6403.043154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6403.044035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.046443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6403.049403] Lustre: Skipped 1 previous similar message [ 6403.053101] Lustre: Skipped 2 previous similar messages [ 6407.278272] LustreError: 354217:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6407.280535] LustreError: 354217:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6407.326695] Lustre: server umount lustre-OST0000 complete [ 6410.208886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6410.211629] LustreError: Skipped 1 previous similar message [ 6410.213231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.219015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6414.667809] LustreError: 354420:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6414.670374] LustreError: 354420:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6414.702888] Lustre: server umount lustre-MDT0000 complete [ 6415.987323] LustreError: 351269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355911 with bad export cookie 13680301601475708369 [ 6415.990645] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.991971] LustreError: 351269:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6416.108713] Lustre: server umount lustre-MDT0001 complete [ 6418.973074] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:05:13 (1763355913) [ 6421.359406] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:05:16 (1763355916) [ 6424.144355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6424.322629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6425.530209] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6428.457368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6429.973472] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6431.841638] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6433.087613] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6435.376491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6437.359364] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6439.129772] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6452.884578] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6453.478537] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6454.071116] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6454.636299] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6455.296212] Lustre: Failing over lustre-MDT0000 [ 6455.422777] LustreError: 358093:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6455.425584] LustreError: 358093:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6455.456808] Lustre: server umount lustre-MDT0000 complete [ 6459.364502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.365709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6459.366754] LustreError: 357787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6459.366769] LustreError: 357787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6459.368981] Lustre: Skipped 3 previous similar messages [ 6464.481472] LustreError: 355633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6464.485979] LustreError: 355633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6468.326811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.370541] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6468.472114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6468.474467] Lustre: Skipped 2 previous similar messages [ 6468.497086] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6469.734205] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6469.893317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6473.706406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6473.713064] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6473.729340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6474.845626] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6475.357596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6477.171075] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6477.781433] Lustre: Failing over lustre-MDT0000 [ 6477.954424] LustreError: 359169:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6477.956158] LustreError: 359169:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6477.991681] Lustre: server umount lustre-MDT0000 complete [ 6478.818429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6478.818503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.820059] LustreError: 355629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.820069] LustreError: 355629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6478.832478] Lustre: Skipped 1 previous similar message [ 6483.937223] LustreError: 355651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.942944] LustreError: 355651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6489.057375] LustreError: 355651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.062623] LustreError: 355651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6491.105518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.149957] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6491.289452] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6492.567638] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6495.492720] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6496.743287] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6496.745080] Lustre: Skipped 2 previous similar messages [ 6496.751592] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6496.769289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6497.833211] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6498.335717] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6501.859620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6501.860314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6501.861703] Lustre: Skipped 4 previous similar messages [ 6506.643957] Lustre: server umount lustre-MDT0000 complete [ 6506.977421] LustreError: 355632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6506.982286] LustreError: 355632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6507.830070] LustreError: 355614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356003 with bad export cookie 13680301601475711043 [ 6507.832540] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6507.833085] LustreError: 355614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6507.941501] Lustre: server umount lustre-MDT0001 complete [ 6519.346718] LustreError: 360602:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6519.349425] LustreError: 360602:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6519.397355] Lustre: server umount lustre-OST0000 complete [ 6524.897972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6525.071172] LustreError: 361430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6526.496521] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6529.565864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6531.202549] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6533.101952] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6534.388530] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6536.767795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6536.901593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.903785] Lustre: Skipped 3 previous similar messages [ 6538.873028] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6540.776924] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6550.247455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6555.619314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.619663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6555.625551] Lustre: Skipped 4 previous similar messages [ 6555.629696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6555.632741] Lustre: Skipped 1 previous similar message [ 6558.675845] Lustre: server umount lustre-OST0000 complete [ 6566.107953] Lustre: server umount lustre-MDT0000 complete [ 6567.404297] LustreError: 361410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356062 with bad export cookie 13680301601475711750 [ 6567.407413] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6567.408089] LustreError: 361410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6567.529026] Lustre: server umount lustre-MDT0001 complete [ 6571.163259] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6572.239271] Key type lgssc unregistered [ 6572.378413] LNet: 364630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6572.382151] LNetError: 364630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6572.392292] LNet: Removed LNI 192.168.201.116@tcp [ 6572.701806] Key type .llcrypt unregistered [ 6572.703585] Key type ._llcrypt unregistered [ 6582.718331] Key type ._llcrypt registered [ 6582.719537] Key type .llcrypt registered [ 6582.769966] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6583.211942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6583.277855] alg: No test for adler32 (adler32-zlib) [ 6584.153139] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6584.248671] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6585.840148] Key type lgssc registered [ 6586.227278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6589.081055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6589.087744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.258517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6591.720628] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6594.833391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6594.970208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6596.425864] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6598.313164] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6599.565322] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6601.888236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6602.024351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6604.086375] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6605.947392] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6615.342299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6620.642420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6620.644529] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6620.646365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6620.648686] Lustre: Skipped 2 previous similar messages [ 6620.650423] Lustre: Skipped 1 previous similar message [ 6625.763474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6625.765684] Lustre: Skipped 3 previous similar messages [ 6626.433313] LustreError: 368371:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6626.469175] Lustre: server umount lustre-MDT0000 complete [ 6627.788927] LustreError: 366085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356123 with bad export cookie 6665527958244684831 [ 6627.793200] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.794456] LustreError: 366085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6627.871994] LustreError: 368574:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6627.874717] LustreError: 368574:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6627.932884] Lustre: server umount lustre-MDT0001 complete [ 6639.539858] LustreError: 368774:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6639.541467] LustreError: 368774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6639.587549] Lustre: server umount lustre-OST0000 complete [ 6644.352447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.527763] LustreError: 369505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.536028] LustreError: 369505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6644.552142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6646.006863] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6649.053608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6649.189244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6650.597625] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6652.515927] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6653.785716] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6656.153164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6658.225412] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6660.078197] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6668.587437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6679.011017] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.012180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6679.015169] Lustre: Skipped 2 previous similar messages [ 6679.017554] Lustre: Skipped 1 previous similar message [ 6683.712481] LustreError: 371876:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6683.716396] LustreError: 371876:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6683.750267] Lustre: server umount lustre-MDT0000 complete [ 6684.129887] LustreError: 369505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.135147] LustreError: 369505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6685.133337] LustreError: 369486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356180 with bad export cookie 6665527958244685930 [ 6685.136344] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6685.138509] LustreError: 369486:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6685.257601] Lustre: server umount lustre-MDT0001 complete [ 6696.885392] LustreError: 372278:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6696.887541] LustreError: 372278:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6696.935544] Lustre: server umount lustre-OST0000 complete [ 6702.447488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.618512] LustreError: 373108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6702.642223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6702.644545] Lustre: Skipped 1 previous similar message [ 6704.071856] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6707.070400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6708.568845] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6710.424535] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6711.595101] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6713.836973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6715.871742] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6717.696931] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6728.300921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6732.769786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6732.774205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.780563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6732.784058] Lustre: Skipped 1 previous similar message [ 6733.796245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6735.854093] LustreError: 375371:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6735.855774] LustreError: 375371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6735.889121] Lustre: server umount lustre-OST0000 complete [ 6737.889236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6737.891292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.894402] Lustre: Skipped 1 previous similar message [ 6737.896117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6737.897929] Lustre: Skipped 2 previous similar messages [ 6743.320584] Lustre: server umount lustre-MDT0000 complete [ 6744.033465] LustreError: 373108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.040220] LustreError: 373108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6744.589425] LustreError: 373088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356240 with bad export cookie 6665527958244686938 [ 6744.592569] LustreError: 373088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6744.593862] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.726647] Lustre: server umount lustre-MDT0001 complete [ 6748.330392] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6749.455421] Key type lgssc unregistered [ 6749.590423] LNet: 376306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6749.592581] LNetError: 376306:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6749.601441] LNet: Removed LNI 192.168.201.116@tcp [ 6749.915109] Key type .llcrypt unregistered [ 6749.916140] Key type ._llcrypt unregistered [ 6759.303917] Key type ._llcrypt registered [ 6759.305513] Key type .llcrypt registered [ 6759.344831] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6759.907811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6759.936948] alg: No test for adler32 (adler32-zlib) [ 6760.807502] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6760.913099] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6762.504195] Key type lgssc registered [ 6762.928688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6765.781629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6765.791119] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.793784] LustreError: 377702:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6765.801387] LustreError: 377702:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6765.805242] LustreError: 377702:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6765.809516] LustreError: 377702:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6765.812761] LustreError: 377702:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6768.557632] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6768.561691] LustreError: 378101:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6768.572452] LustreError: 378101:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6768.576439] LustreError: 378101:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6768.580178] LustreError: 378101:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6768.585158] LustreError: 378101:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6775.324431] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6776.516742] Key type lgssc unregistered [ 6776.649484] LNet: 379084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6776.653198] LNetError: 379084:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6776.666454] LNet: Removed LNI 192.168.201.116@tcp [ 6776.992638] Key type .llcrypt unregistered [ 6776.993643] Key type ._llcrypt unregistered [ 6783.377373] Key type ._llcrypt registered [ 6783.378885] Key type .llcrypt registered [ 6783.417740] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6789.721224] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6790.215398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6790.225794] alg: No test for adler32 (adler32-zlib) [ 6791.090866] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6791.181678] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6792.768182] Key type lgssc registered [ 6793.149484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6796.084558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6798.237871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6800.301166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6802.366459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6804.644938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6804.667235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6804.672569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6805.759834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6805.770810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6805.811114] Lustre: lustre-MDT0000: new disk, initializing [ 6805.836943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6805.843982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6807.211429] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6811.347269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6811.374173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6811.404891] Lustre: 382187:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6811.426776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6811.428241] Lustre: Skipped 1 previous similar message [ 6811.477895] Lustre: lustre-MDT0001: new disk, initializing [ 6811.514597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6811.529836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6811.534192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6812.909026] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6815.962539] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6817.115966] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6819.416849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6819.439055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6819.540854] Lustre: lustre-OST0000: new disk, initializing [ 6819.543289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6819.571586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6821.475645] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6824.548119] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6825.946946] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6828.026615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6828.030418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6828.061958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6829.074648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6830.384342] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6830.456167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6833.124048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6833.124320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6833.127859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6833.128837] LustreError: Skipped 1 previous similar message [ 6833.131453] Lustre: Skipped 1 previous similar message [ 6837.167072] LustreError: 385151:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6837.206461] Lustre: server umount lustre-OST0000 complete [ 6842.336872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6842.340340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.345278] Lustre: Skipped 1 previous similar message [ 6842.347751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6843.361141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6843.361670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.542522] LustreError: 385352:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6844.544421] LustreError: 385352:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6844.571554] Lustre: server umount lustre-MDT0000 complete [ 6845.829424] LustreError: 382180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356341 with bad export cookie 3230354586602567886 [ 6845.831848] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.837611] LustreError: 382180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6845.912571] LustreError: 385553:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6845.914905] LustreError: 385553:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6845.968926] Lustre: server umount lustre-MDT0001 complete [ 6848.835285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6849.390180] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:12:24 (1763356344) [ 6851.695880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.881752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.252924] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6856.191088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6856.334370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6857.658038] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6859.363396] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6860.529518] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6862.761363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6862.821244] LustreError: 387865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.825540] LustreError: 387865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6864.614994] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6866.275875] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6868.516679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6868.540984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6868.598425] Lustre: lustre-OST0001: new disk, initializing [ 6868.600036] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6868.634539] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6868.636610] Lustre: Skipped 1 previous similar message [ 6870.409432] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6873.387812] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6875.135530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6875.139379] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6875.167798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6875.170176] LustreError: lustre-OST0001-osc-MDT0000: operation ost_create to node 0@lo failed: rc = -107 [ 6875.171898] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.175378] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6875.176604] Lustre: Skipped 1 previous similar message [ 6880.014531] Lustre: lustre-OST0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 6880.225709] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6880.227758] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6881.262148] LustreError: 389471:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6881.263990] LustreError: 389471:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6881.282443] Lustre: server umount lustre-OST0001 complete [ 6885.347355] LustreError: 388373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.354438] LustreError: 388373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6890.468281] LustreError: 387863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6890.475760] LustreError: 387863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6895.587378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.588685] LustreError: 388373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6895.589392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6895.589400] Lustre: Skipped 2 previous similar messages [ 6895.592310] Lustre: Skipped 1 previous similar message [ 6895.599887] LustreError: 388373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6899.375074] LustreError: 389695:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6899.376725] LustreError: 389695:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6899.412774] Lustre: server umount lustre-OST0000 complete [ 6900.654132] LustreError: 389473:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0001-osc-MDT0000: can't precreate: rc = -5 [ 6900.657447] LustreError: 389473:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0001-osc-MDT0000: cannot precreate objects: rc = -5 [ 6906.911862] Lustre: server umount lustre-MDT0000 complete [ 6908.146325] LustreError: 386251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356403 with bad export cookie 3230354586602568740 [ 6908.149884] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6914.304383] LustreError: 390097:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6914.306659] LustreError: 390097:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6914.367499] Lustre: server umount lustre-MDT0001 complete [ 6918.303470] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6919.348234] Key type lgssc unregistered [ 6919.486426] LNet: 390629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6919.488938] LNetError: 390629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6919.500268] LNet: Removed LNI 192.168.201.116@tcp [ 6919.798521] Key type .llcrypt unregistered [ 6919.799910] Key type ._llcrypt unregistered [ 6931.050299] Key type ._llcrypt registered [ 6931.051379] Key type .llcrypt registered [ 6931.096724] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6937.587641] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6938.066092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6938.085343] alg: No test for adler32 (adler32-zlib) [ 6938.944050] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 6939.030044] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6940.616133] Key type lgssc registered [ 6941.004080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6944.060156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6946.302371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6948.463863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6950.709921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6953.095372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6953.112485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6953.117576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.209344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6954.222345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6954.261344] Lustre: lustre-MDT0000: new disk, initializing [ 6954.287228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6954.293642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6955.671211] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6959.955803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6959.981428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6960.018539] Lustre: 394549:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6960.041501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6960.043118] Lustre: Skipped 1 previous similar message [ 6960.083293] Lustre: lustre-MDT0001: new disk, initializing [ 6960.122957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6960.139038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6960.142503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6961.633970] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6964.921615] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6966.190571] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6968.508182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6968.532543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6968.647412] Lustre: lustre-OST0000: new disk, initializing [ 6968.649333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6968.679962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.646559] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6973.747964] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6975.169542] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6977.017620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6977.021616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6977.060235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6977.304178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6978.668400] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6978.745058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6982.114525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6982.115084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.117949] LustreError: Skipped 1 previous similar message [ 6982.121776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6982.125200] Lustre: Skipped 1 previous similar message [ 6985.455068] LustreError: 397504:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6985.492202] Lustre: server umount lustre-OST0000 complete [ 6987.234229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.235432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6987.237120] Lustre: Skipped 2 previous similar messages [ 6992.353796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6992.355358] Lustre: Skipped 2 previous similar messages [ 6997.473087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6997.475268] Lustre: Skipped 1 previous similar message [ 7001.056162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7001.101621] LustreError: 397706:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7001.103913] LustreError: 397706:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7001.131183] Lustre: server umount lustre-MDT0000 complete [ 7002.433141] LustreError: 394542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356497 with bad export cookie 16703148785424355034 [ 7002.436459] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7002.437918] LustreError: 394542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7002.508795] LustreError: 397909:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7002.510582] LustreError: 397909:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7002.576620] Lustre: server umount lustre-MDT0001 complete [ 7005.181786] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 00:14:59 (1763356499) [ 7008.139930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7010.782929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.931624] Lustre: lustre-MDT0000: new disk, initializing [ 7010.942850] LustreError: 399176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.949523] LustreError: 399176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7010.967242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.252000] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7015.205468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7015.341847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7016.765184] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7018.603547] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7019.863928] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7024.674115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.674931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7024.678771] Lustre: Skipped 1 previous similar message [ 7026.749797] LustreError: 400520:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7026.751779] LustreError: 400520:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7026.778336] Lustre: server umount lustre-MDT0000 complete [ 7028.132521] LustreError: 400523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356523 with bad export cookie 16703148785424355902 [ 7028.135456] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7028.136645] LustreError: 400523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7028.252311] Lustre: server umount lustre-MDT0001 complete [ 7031.562347] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:15:26 (1763356526) [ 7033.957634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.128230] LustreError: 401441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.160284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.425566] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7038.167070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7039.593212] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7041.276884] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7042.369381] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7044.482564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7046.338971] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7048.109206] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7051.842787] Lustre: Permanently deactivating lustre-OST0000 [ 7051.847843] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7053.041793] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7053.818709] LustreError: 403939:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7053.823093] LustreError: 403939:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7053.824995] LustreError: 403939:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7054.467853] LustreError: 403987:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7054.470585] LustreError: 403987:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7061.166083] LustreError: 404084:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7061.167770] LustreError: 404084:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7061.204530] Lustre: server umount lustre-OST0000 complete [ 7062.059641] LustreError: 404236:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 7062.061379] LustreError: 404236:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7063.009281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.009787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.012758] Lustre: Skipped 1 previous similar message [ 7063.014115] Lustre: Skipped 1 previous similar message [ 7073.253298] LustreError: 404333:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7073.255167] LustreError: 404333:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7073.278926] Lustre: server umount lustre-MDT0000 complete [ 7074.527451] LustreError: 401422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356569 with bad export cookie 16703148785424356441 [ 7074.531168] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7074.532184] LustreError: 401422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7084.056569] Lustre: server umount lustre-MDT0001 complete [ 7086.721921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.913879] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7088.753511] LustreError: 405376:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7090.753521] Lustre: 405714:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.116@tcp [ 7091.386808] LustreError: 405811:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7091.389363] LustreError: 405811:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7091.417403] Lustre: server umount MGS complete [ 7094.980093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.126762] LustreError: 406439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7095.131690] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7095.135174] Lustre: Skipped 1 previous similar message [ 7095.156170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7095.158502] Lustre: Skipped 2 previous similar messages [ 7096.424227] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7099.221030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7099.318710] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7100.608145] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7102.400650] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7103.615441] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7104.256591] Lustre: Permanently reactivating lustre-OST0000 [ 7104.261601] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7104.264295] Lustre: Skipped 2 previous similar messages [ 7132.128278] LustreError: 408092:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7135.670989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7135.807332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.809242] Lustre: Skipped 1 previous similar message [ 7137.751712] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7139.683116] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7152.610568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7152.613055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7152.613974] LustreError: Skipped 1 previous similar message [ 7152.619990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7152.623222] Lustre: Skipped 6 previous similar messages [ 7153.774099] LustreError: 409918:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7153.776552] LustreError: 409918:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7153.816895] Lustre: server umount lustre-OST0000 complete [ 7155.680885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7157.331992] Lustre: server umount lustre-MDT0000 complete [ 7157.729241] LustreError: 406438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.733602] LustreError: 406438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7158.576437] LustreError: 406419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356654 with bad export cookie 16703148785424357372 [ 7158.578532] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7158.579445] LustreError: 406419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7158.701348] Lustre: server umount lustre-MDT0001 complete [ 7162.369606] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7163.578619] Key type lgssc unregistered [ 7163.726384] LNet: 410854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7163.728439] LNetError: 410854:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7163.735296] LNet: Removed LNI 192.168.201.116@tcp [ 7164.059071] Key type .llcrypt unregistered [ 7164.060662] Key type ._llcrypt unregistered [ 7170.248912] Key type ._llcrypt registered [ 7170.249882] Key type .llcrypt registered [ 7170.289987] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7176.439991] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7176.843968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7176.890406] alg: No test for adler32 (adler32-zlib) [ 7177.752658] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 7177.841949] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7179.424180] Key type lgssc registered [ 7179.805556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7182.767079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7185.132814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7187.300548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7189.494547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7191.925914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7191.947709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7191.954501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.039500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7193.052682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7193.091779] Lustre: lustre-MDT0000: new disk, initializing [ 7193.116235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7193.124217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7194.467424] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7198.726254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7198.748620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7198.790711] Lustre: 413972:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7198.816907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7198.820057] Lustre: Skipped 1 previous similar message [ 7198.859546] Lustre: lustre-MDT0001: new disk, initializing [ 7198.895599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7198.908791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7198.912955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7200.266216] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7203.492965] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7204.741919] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7207.012381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7207.036395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7207.139120] Lustre: lustre-OST0000: new disk, initializing [ 7207.140912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7207.171707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7209.146052] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7212.276587] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7213.048465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7213.053142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7213.091047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7213.715855] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7213.793886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7215.176492] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7215.259843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7218.146138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7218.148405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7218.149617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7218.155133] Lustre: Skipped 1 previous similar message [ 7222.000083] LustreError: 416911:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7222.044332] Lustre: server umount lustre-OST0000 complete [ 7224.801277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7224.804473] LustreError: Skipped 1 previous similar message [ 7224.806937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7224.813328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7224.815706] Lustre: Skipped 1 previous similar message [ 7228.385106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7228.385738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.502265] LustreError: 417112:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7229.504635] LustreError: 417112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7229.530835] Lustre: server umount lustre-MDT0000 complete [ 7230.734055] LustreError: 414753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356726 with bad export cookie 3365391922392513853 [ 7230.736350] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7230.737067] LustreError: 414753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7230.798646] LustreError: 417314:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7230.800188] LustreError: 417314:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7230.841588] Lustre: server umount lustre-MDT0001 complete [ 7233.501843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7234.024165] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:18:48 (1763356728) [ 7236.217444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.386463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7237.580246] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7240.231836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7241.572456] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7243.221208] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7244.331958] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7246.456702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7246.579302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.582396] Lustre: Skipped 1 previous similar message [ 7248.570275] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7250.401164] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7251.061736] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7252.537666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7255.010778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7255.012059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.015346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.018244] Lustre: Skipped 1 previous similar message [ 7255.021693] Lustre: Skipped 1 previous similar message [ 7259.951084] LustreError: 420472:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7259.953070] LustreError: 420472:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7259.986443] Lustre: server umount lustre-OST0000 complete [ 7265.249768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7265.250504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7265.253228] Lustre: Skipped 1 previous similar message [ 7265.256175] Lustre: Skipped 2 previous similar messages [ 7267.389788] LustreError: 420673:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7267.391466] LustreError: 420673:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7267.416835] Lustre: server umount lustre-MDT0000 complete [ 7268.658268] LustreError: 419652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356764 with bad export cookie 3365391922392514721 [ 7268.661040] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7268.662184] LustreError: 419652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7268.772507] Lustre: server umount lustre-MDT0001 complete [ 7272.460542] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7273.474592] Key type lgssc unregistered [ 7273.605305] LNet: 421405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7273.609286] LNetError: 421405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7273.618366] LNet: Removed LNI 192.168.201.116@tcp [ 7273.918655] Key type .llcrypt unregistered [ 7273.919780] Key type ._llcrypt unregistered [ 7281.943974] Key type ._llcrypt registered [ 7281.944904] Key type .llcrypt registered [ 7281.986339] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7282.092376] Key type .llcrypt unregistered [ 7282.094050] Key type ._llcrypt unregistered [ 7289.527244] Key type ._llcrypt registered [ 7289.528911] Key type .llcrypt registered [ 7289.573980] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7290.042721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7290.096951] alg: No test for adler32 (adler32-zlib) [ 7290.977163] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 7291.078370] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7292.672161] Key type lgssc registered [ 7293.055394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7295.844043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7295.849900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.008064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.345604] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7301.230351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7301.355954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.324678] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7305.167862] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7306.721779] LustreError: 423554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.728472] LustreError: 423554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7307.515933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7307.638683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7308.648947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7309.012994] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7312.865289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7312.867636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7312.871760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7313.891408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7313.893020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.893023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7313.967091] LustreError: 425612:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7314.002431] Lustre: server umount lustre-OST0000 complete [ 7316.963357] LustreError: 425525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356812 with bad export cookie 12335687322402170249 [ 7316.964529] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7316.966607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7316.972588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7316.974879] Lustre: Skipped 1 previous similar message [ 7317.507782] LustreError: 425813:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7317.511271] LustreError: 425813:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7317.542276] Lustre: server umount lustre-MDT0000 complete [ 7318.783257] LustreError: 425525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356814 with bad export cookie 12335687322402169759 [ 7318.786725] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7318.852933] LustreError: 426016:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7318.854480] LustreError: 426016:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7318.907291] Lustre: server umount lustre-MDT0001 complete [ 7322.566295] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7323.601336] Key type lgssc unregistered [ 7323.727407] LNet: 426546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7323.729357] LNetError: 426546:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7323.738360] LNet: Removed LNI 192.168.201.116@tcp [ 7324.058601] Key type .llcrypt unregistered [ 7324.060148] Key type ._llcrypt unregistered [ 7333.893286] Key type ._llcrypt registered [ 7333.894838] Key type .llcrypt registered [ 7333.937760] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7334.253443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7334.260770] alg: No test for adler32 (adler32-zlib) [ 7335.117843] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 7335.211071] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7336.792186] Key type lgssc registered [ 7337.195581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7339.868900] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7339.874708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7355.296393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7360.416299] LustreError: 427893:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.116@tcp: failed processing log, type 4: rc = -110 [ 7387.040190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7388.895806] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7390.680589] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7392.972769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.167126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.187838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7394.509949] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7397.450954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7397.568386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7398.906290] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7400.699584] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7401.990318] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7407.076436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.082522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7407.085114] Lustre: Skipped 1 previous similar message [ 7410.415066] LustreError: 430175:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7410.456070] Lustre: server umount lustre-OST0000 complete [ 7412.193530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.199125] Lustre: Skipped 1 previous similar message [ 7412.201916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.203711] Lustre: Skipped 1 previous similar message [ 7417.313687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7417.317029] Lustre: Skipped 1 previous similar message [ 7417.859299] LustreError: 430377:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7417.861390] LustreError: 430377:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7417.896469] Lustre: server umount lustre-MDT0000 complete [ 7419.204552] LustreError: 430088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356914 with bad export cookie 11131509478958975702 [ 7419.207301] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.208841] LustreError: 430088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7419.281418] LustreError: 430581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7419.283041] LustreError: 430581:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7419.343371] Lustre: server umount lustre-MDT0001 complete [ 7423.234605] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7424.355217] Key type lgssc unregistered [ 7424.487406] LNet: 431111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7424.489677] LNetError: 431111:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7424.497316] LNet: Removed LNI 192.168.201.116@tcp [ 7424.836820] Key type .llcrypt unregistered [ 7424.838567] Key type ._llcrypt unregistered [ 7434.784954] Key type ._llcrypt registered [ 7434.785890] Key type .llcrypt registered [ 7434.827610] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7435.199994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7435.215217] alg: No test for adler32 (adler32-zlib) [ 7436.066736] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 7436.144625] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7437.720147] Key type lgssc registered [ 7438.155749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7441.014212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7441.020216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7442.190967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7443.572400] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7446.487577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7446.623067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7447.983619] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7449.756690] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7450.957977] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7453.273968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7453.400494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.423691] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7457.333303] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7466.728412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7472.099214] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7472.099247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7472.100264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7472.104631] Lustre: Skipped 1 previous similar message [ 7472.909098] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 7472.910948] Lustre: Skipped 2 previous similar messages [ 7474.817989] LustreError: 434754:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7474.847078] Lustre: server umount lustre-MDT0000 complete [ 7475.605587] LustreError: 432465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.16@tcp arrived at 1763356971 with bad export cookie 14443048609190879365 [ 7475.608877] LustreError: 432465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7476.258027] LustreError: 432465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356971 with bad export cookie 14443048609190878721 [ 7476.262712] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7477.217547] LustreError: 432485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7477.218150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.223610] LustreError: 432485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7477.228505] Lustre: Skipped 1 previous similar message [ 7477.229203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7481.248807] LustreError: 432485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.249138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7481.257491] LustreError: 432485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7482.351079] LustreError: 434957:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7482.352792] LustreError: 434957:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7482.385905] Lustre: server umount lustre-OST0000 complete [ 7484.601160] LustreError: 435311:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7484.602778] LustreError: 435311:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7484.656281] Lustre: server umount lustre-MDT0001 complete [ 7488.227621] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7489.248557] Key type lgssc unregistered [ 7489.377398] LNet: 435841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7489.379411] LNetError: 435841:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7489.386302] LNet: Removed LNI 192.168.201.116@tcp [ 7489.706931] Key type .llcrypt unregistered [ 7489.708896] Key type ._llcrypt unregistered [ 7498.779783] Key type ._llcrypt registered [ 7498.780836] Key type .llcrypt registered [ 7498.820450] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7499.172680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7499.212078] alg: No test for adler32 (adler32-zlib) [ 7500.061556] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 7500.141644] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7501.720141] Key type lgssc registered [ 7502.075895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7504.623531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7504.628872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.791444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7507.059087] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7509.969348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7510.094559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7511.440153] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7513.151768] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7514.407436] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7516.649176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7516.760691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.671530] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7520.409144] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7531.112219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7540.705070] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7540.707066] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.710767] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7541.732063] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7541.732919] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.733488] Lustre: Skipped 1 previous similar message [ 7542.975869] LustreError: 439482:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7543.000313] Lustre: server umount lustre-MDT0001 complete [ 7546.850479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7546.850551] LustreError: 437939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.850842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7546.851473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7546.874319] LustreError: 437939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7550.765065] LustreError: 439686:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7550.767400] LustreError: 439686:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7550.804702] Lustre: server umount lustre-OST0000 complete [ 7551.970744] LustreError: 437939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7552.171738] LustreError: 439889:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7552.173400] LustreError: 439889:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7552.238100] Lustre: server umount lustre-MDT0000 complete [ 7556.947027] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7558.010266] Key type lgssc unregistered [ 7558.152495] LNet: 440572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7558.155239] LNetError: 440572:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7558.165363] LNet: Removed LNI 192.168.201.116@tcp [ 7558.504384] Key type .llcrypt unregistered [ 7558.505775] Key type ._llcrypt unregistered [ 7567.184581] Key type ._llcrypt registered [ 7567.185573] Key type .llcrypt registered [ 7567.225310] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7573.335849] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7573.662048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7573.670686] alg: No test for adler32 (adler32-zlib) [ 7574.539896] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 7574.629361] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7576.208180] Key type lgssc registered [ 7576.574783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7579.535813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7581.666462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7583.661828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7585.855441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7589.229193] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7592.375784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7594.672975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7594.693414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7594.698675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7595.788203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7595.801222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7595.847739] Lustre: lustre-MDT0000: new disk, initializing [ 7595.875498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.883160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7597.207219] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7600.680271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7602.925214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7602.949570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7603.048569] Lustre: lustre-OST0000: new disk, initializing [ 7603.050284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7603.051649] Lustre: Skipped 1 previous similar message [ 7603.073836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7605.001727] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7608.111095] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7611.389882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7611.393291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7611.405949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7616.889600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7619.295038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7619.318042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7619.356080] Lustre: 444517:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7619.392348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7619.429265] Lustre: lustre-MDT0001: new disk, initializing [ 7619.510601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7619.532663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7619.538442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7620.937139] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7624.623706] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7624.706413] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7626.191305] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7626.286787] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7629.797888] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7629.798102] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.801408] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7629.805505] Lustre: Skipped 1 previous similar message [ 7629.808555] Lustre: Skipped 1 previous similar message [ 7634.112834] LustreError: 447941:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7634.134697] Lustre: server umount lustre-MDT0001 complete [ 7640.034358] LustreError: 445534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7640.042081] LustreError: 445534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7645.154263] LustreError: 444525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7645.159086] LustreError: 444525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7647.200802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7647.202628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.205777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7647.207161] Lustre: Skipped 1 previous similar message [ 7650.273187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7650.273880] LustreError: 446763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7652.526103] LustreError: 448213:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7652.527838] LustreError: 448213:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7652.562338] Lustre: server umount lustre-OST0000 complete [ 7660.094036] LustreError: 448414:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7660.095552] LustreError: 448414:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7660.170792] Lustre: server umount lustre-MDT0000 complete [ 7664.884220] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7665.967106] Key type lgssc unregistered [ 7666.109465] LNet: 449100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7666.112030] LNetError: 449100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7666.119415] LNet: Removed LNI 192.168.201.116@tcp [ 7666.427616] Key type .llcrypt unregistered [ 7666.429292] Key type ._llcrypt unregistered [ 7679.668339] Key type ._llcrypt registered [ 7679.669601] Key type .llcrypt registered [ 7679.707632] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7679.809527] Key type .llcrypt unregistered [ 7679.811383] Key type ._llcrypt unregistered [ 7686.541647] Key type ._llcrypt registered [ 7686.544072] Key type .llcrypt registered [ 7686.588855] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7687.033847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7687.040391] alg: No test for adler32 (adler32-zlib) [ 7687.915638] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 7688.015575] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7689.608208] Key type lgssc registered [ 7690.027076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7692.789103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7694.895182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7696.998657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7699.266334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7699.283557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7699.287626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.372540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7700.384768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7700.421901] Lustre: lustre-MDT0000: new disk, initializing [ 7700.447524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.453988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7701.880861] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7706.011301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7706.031929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7706.064542] Lustre: 452417:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7706.088849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7706.090884] Lustre: Skipped 1 previous similar message [ 7706.126492] Lustre: lustre-MDT0001: new disk, initializing [ 7706.161468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7706.175546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7706.178233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7707.456683] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7710.568352] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7711.940549] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7714.336635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7714.359314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7714.448829] Lustre: lustre-OST0000: new disk, initializing [ 7714.450513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7714.477702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.443386] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7716.668138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7716.673044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7716.706540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7719.582993] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7726.561172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7726.563317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.567148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7727.075796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7727.077437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7727.080834] Lustre: Skipped 1 previous similar message [ 7728.127578] LustreError: 454858:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7728.160498] Lustre: server umount lustre-MDT0000 complete [ 7729.358122] LustreError: 452410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357224 with bad export cookie 4906654102871991307 [ 7729.362058] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7729.362478] LustreError: 452410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7729.428210] LustreError: 455059:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7729.431487] LustreError: 455059:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7729.472591] Lustre: server umount lustre-MDT0001 complete [ 7740.914344] LustreError: 455260:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7740.918021] LustreError: 455260:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7740.969038] Lustre: server umount lustre-OST0000 complete [ 7744.354569] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:27:19 (1763357239) [ 7744.694728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7746.936767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.113205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7748.460489] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7751.115316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7751.234584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7752.531285] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7754.274640] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7755.471326] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7757.625573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7757.651501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7757.719942] Lustre: Found index 0 for lustre-OST0000, updating log [ 7759.619754] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7761.315892] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7772.138709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7781.344746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7781.346528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.349535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7781.350990] Lustre: Skipped 2 previous similar messages [ 7784.367077] LustreError: 458368:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7784.369354] LustreError: 458368:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7784.408213] Lustre: server umount lustre-OST0000 complete [ 7785.303206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7786.976988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7786.979056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.983906] Lustre: Skipped 1 previous similar message [ 7786.985535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.988014] Lustre: Skipped 3 previous similar messages [ 7793.633953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7793.637360] Lustre: Skipped 5 previous similar messages [ 7800.288171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7800.327013] LustreError: 458623:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7800.330624] LustreError: 458623:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7800.355800] Lustre: server umount lustre-MDT0000 complete [ 7801.567951] LustreError: 456065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357297 with bad export cookie 4906654102871992686 [ 7801.570565] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7801.570686] LustreError: 456065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7801.681427] Lustre: server umount lustre-MDT0001 complete [ 7804.541918] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:28:19 (1763357299) [ 7804.908508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7805.271605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7805.593564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7805.958912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7806.284926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7806.630184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7806.965084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7807.315800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7807.661441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7807.997266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7808.321421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7808.677635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7809.004664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7809.339683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7809.761859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7810.090313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7810.420216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7810.742198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7811.048262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7811.381602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7811.779800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7812.229817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7812.734490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7813.152232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7813.504140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7813.825317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7814.138840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7814.515145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7814.851384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7815.198447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7815.553201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7815.905375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7816.325470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7816.746686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7817.192488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7817.591576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7817.930151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7818.306310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7818.680025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7819.055071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7819.454070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7819.810159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7820.159037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7820.500024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7820.825806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7821.157703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7821.495553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7821.831593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7822.162696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7822.499970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7822.815295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7824.224187] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp [ 7825.202122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7825.365165] LustreError: 462209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7825.369320] LustreError: 462209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7825.387389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7825.389848] Lustre: Skipped 1 previous similar message [ 7826.596579] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7829.595324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7830.906325] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7832.591405] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7833.779278] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7835.892419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7835.923265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7836.002977] Lustre: Found index 0 for lustre-OST0000, updating log [ 7837.902557] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7839.617760] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7840.412153] LustreError: 464516:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7840.413879] LustreError: 464516:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7840.442989] Lustre: server umount lustre-OST0000 complete [ 7845.344983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7845.349042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7845.353339] Lustre: Skipped 1 previous similar message [ 7845.355204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7845.356805] Lustre: Skipped 2 previous similar messages [ 7848.091073] Lustre: server umount lustre-MDT0000 complete [ 7849.446761] LustreError: 463887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357344 with bad export cookie 4906654102871993757 [ 7849.453268] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7849.453610] LustreError: 463887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7849.579164] Lustre: server umount lustre-MDT0001 complete [ 7850.664861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7856.247032] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7859.662361] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7863.106230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7865.215409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7867.308048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7869.453460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7871.781281] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:29:26 (1763357366) [ 7886.688208] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp [ 7894.083487] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7895.074238] Key type lgssc unregistered [ 7898.272669] LNet: 468510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7898.276487] LNetError: 468510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8025.257100] LNet: Removed LNI 192.168.201.116@tcp [ 8025.745467] Key type .llcrypt unregistered [ 8025.747302] Key type ._llcrypt unregistered [ 8033.308731] Key type ._llcrypt registered [ 8033.310010] Key type .llcrypt registered [ 8033.353120] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8033.649770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8033.686061] alg: No test for adler32 (adler32-zlib) [ 8034.548933] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 8034.638991] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8036.232199] Key type lgssc registered [ 8036.682604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8038.567981] LNet: Added LNI 192.168.201.116@tcp100 [8/256/0/180] [ 8039.017509] LNet: Added LNI 192.168.201.116@tcp101 [8/256/0/180] [ 8039.357766] LNet: Added LNI 192.168.201.116@tcp102 [8/256/0/180] [ 8039.785731] LNet: Added LNI 192.168.201.116@tcp103 [8/256/0/180] [ 8040.140798] LNet: Added LNI 192.168.201.116@tcp104 [8/256/0/180] [ 8040.461070] LNet: Added LNI 192.168.201.116@tcp105 [8/256/0/180] [ 8040.803198] LNet: Added LNI 192.168.201.116@tcp106 [8/256/0/180] [ 8041.197853] LNet: Added LNI 192.168.201.116@tcp107 [8/256/0/180] [ 8041.505502] LNet: Added LNI 192.168.201.116@tcp108 [8/256/0/180] [ 8041.813220] LNet: Added LNI 192.168.201.116@tcp109 [8/256/0/180] [ 8042.205907] LNet: Added LNI 192.168.201.116@tcp110 [8/256/0/180] [ 8042.558815] LNet: Added LNI 192.168.201.116@tcp111 [8/256/0/180] [ 8042.870471] LNet: Added LNI 192.168.201.116@tcp112 [8/256/0/180] [ 8043.200832] LNet: Added LNI 192.168.201.116@tcp113 [8/256/0/180] [ 8043.574904] LNet: Added LNI 192.168.201.116@tcp114 [8/256/0/180] [ 8044.024853] LNet: Added LNI 192.168.201.116@tcp115 [8/256/0/180] [ 8044.438096] LNet: Added LNI 192.168.201.116@tcp116 [8/256/0/180] [ 8044.854938] LNet: Added LNI 192.168.201.116@tcp117 [8/256/0/180] [ 8045.264796] LNet: Added LNI 192.168.201.116@tcp118 [8/256/0/180] [ 8045.695857] LNet: Added LNI 192.168.201.116@tcp119 [8/256/0/180] [ 8046.087360] LNet: Added LNI 192.168.201.116@tcp120 [8/256/0/180] [ 8046.469234] LNet: Added LNI 192.168.201.116@tcp121 [8/256/0/180] [ 8046.756374] LNet: Added LNI 192.168.201.116@tcp122 [8/256/0/180] [ 8047.068990] LNet: Added LNI 192.168.201.116@tcp123 [8/256/0/180] [ 8047.429420] LNet: Added LNI 192.168.201.116@tcp124 [8/256/0/180] [ 8047.746830] LNet: Added LNI 192.168.201.116@tcp125 [8/256/0/180] [ 8048.132465] LNet: Added LNI 192.168.201.116@tcp126 [8/256/0/180] [ 8048.508749] LNet: Added LNI 192.168.201.116@tcp127 [8/256/0/180] [ 8048.902113] LNet: Added LNI 192.168.201.116@tcp128 [8/256/0/180] [ 8049.315615] LNet: Added LNI 192.168.201.116@tcp129 [8/256/0/180] [ 8049.744169] LNet: Added LNI 192.168.201.116@tcp130 [8/256/0/180] [ 8050.079127] LNet: Added LNI 192.168.201.116@tcp131 [8/256/0/180] [ 8050.499250] LNet: Added LNI 192.168.201.116@tcp132 [8/256/0/180] [ 8050.911539] LNet: Added LNI 192.168.201.116@tcp133 [8/256/0/180] [ 8051.305423] LNet: Added LNI 192.168.201.116@tcp134 [8/256/0/180] [ 8051.647923] LNet: Added LNI 192.168.201.116@tcp135 [8/256/0/180] [ 8054.558753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8054.585475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8054.593452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.642971] Lustre: 471499:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 8055.714626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8055.726747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8055.768375] Lustre: lustre-MDT0000: new disk, initializing [ 8055.801337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8055.809584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8057.403300] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8062.905576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8062.942725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8063.018215] Lustre: 471517:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8063.056926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8063.058676] Lustre: Skipped 1 previous similar message [ 8063.098890] Lustre: lustre-MDT0001: new disk, initializing [ 8063.112501] Lustre: 472303:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 8063.115549] Lustre: 472303:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8063.162028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8063.179145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8063.183923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8064.665492] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8065.035537] Lustre: 472587:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 8065.042200] Lustre: 472587:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8067.891661] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8069.147801] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8071.582742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8071.602644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8071.701051] Lustre: lustre-OST0000: new disk, initializing [ 8071.702598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8071.731058] Lustre: 473273:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 8071.733510] Lustre: 473273:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 8071.749234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8074.017175] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8077.825680] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8079.871930] Lustre: 473997:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 8079.878869] Lustre: 473997:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8079.906547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8079.909911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8079.961881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8084.964841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8084.966521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.967055] LustreError: Skipped 1 previous similar message [ 8084.971497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8084.974858] Lustre: Skipped 1 previous similar message [ 8086.899123] LustreError: 474108:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8086.942325] Lustre: server umount lustre-OST0000 complete [ 8089.056941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8089.059225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.062861] Lustre: Skipped 1 previous similar message [ 8089.064683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8090.081453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8090.082272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.337405] LustreError: 474310:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8094.341052] LustreError: 474310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8094.371821] Lustre: server umount lustre-MDT0000 complete [ 8095.531677] LustreError: 471509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357590 with bad export cookie 10956377687250982828 [ 8095.536289] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8095.536884] LustreError: 471509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8095.623273] LustreError: 474513:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8095.625617] LustreError: 474513:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8095.675577] Lustre: server umount lustre-MDT0001 complete [ 8099.350272] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8100.310041] Key type lgssc unregistered [ 8100.450545] LNet: 475053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8100.454442] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8100.457928] LNet: Removed LNI 192.168.201.116@tcp [ 8101.472165] LNet: Removed LNI 192.168.201.116@tcp100 [ 8101.473603] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8101.475363] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8101.477454] LNet: Removed LNI 192.168.201.116@tcp101 [ 8102.496236] LNet: Removed LNI 192.168.201.116@tcp102 [ 8102.498526] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8102.502575] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8102.506968] LNet: Removed LNI 192.168.201.116@tcp103 [ 8103.520217] LNet: Removed LNI 192.168.201.116@tcp104 [ 8104.544121] LNet: Removed LNI 192.168.201.116@tcp105 [ 8104.545554] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8104.547982] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8104.549975] LNet: Removed LNI 192.168.201.116@tcp106 [ 8105.568190] LNet: Removed LNI 192.168.201.116@tcp107 [ 8106.592139] LNet: Removed LNI 192.168.201.116@tcp108 [ 8107.616261] LNet: Removed LNI 192.168.201.116@tcp109 [ 8108.640179] LNet: Removed LNI 192.168.201.116@tcp110 [ 8108.641621] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8108.643771] LNetError: 475053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8108.645925] LNet: Removed LNI 192.168.201.116@tcp111 [ 8109.664186] LNet: Removed LNI 192.168.201.116@tcp112 [ 8110.688173] LNet: Removed LNI 192.168.201.116@tcp113 [ 8111.712208] LNet: Removed LNI 192.168.201.116@tcp114 [ 8111.714340] LNet: Removed LNI 192.168.201.116@tcp115 [ 8111.716402] LNet: Removed LNI 192.168.201.116@tcp116 [ 8111.718552] LNet: Removed LNI 192.168.201.116@tcp117 [ 8111.720362] LNet: Removed LNI 192.168.201.116@tcp118 [ 8111.722306] LNet: Removed LNI 192.168.201.116@tcp119 [ 8111.724123] LNet: Removed LNI 192.168.201.116@tcp120 [ 8111.725856] LNet: Removed LNI 192.168.201.116@tcp121 [ 8111.727788] LNet: Removed LNI 192.168.201.116@tcp122 [ 8111.729767] LNet: Removed LNI 192.168.201.116@tcp123 [ 8111.731665] LNet: Removed LNI 192.168.201.116@tcp124 [ 8111.733645] LNet: Removed LNI 192.168.201.116@tcp125 [ 8111.735671] LNet: Removed LNI 192.168.201.116@tcp126 [ 8111.737820] LNet: Removed LNI 192.168.201.116@tcp127 [ 8111.739829] LNet: Removed LNI 192.168.201.116@tcp128 [ 8111.742070] LNet: Removed LNI 192.168.201.116@tcp129 [ 8111.743554] LNet: Removed LNI 192.168.201.116@tcp130 [ 8111.744671] LNet: Removed LNI 192.168.201.116@tcp131 [ 8111.746070] LNet: Removed LNI 192.168.201.116@tcp132 [ 8111.747557] LNet: Removed LNI 192.168.201.116@tcp133 [ 8111.748966] LNet: Removed LNI 192.168.201.116@tcp134 [ 8111.756746] LNet: Removed LNI 192.168.201.116@tcp135 [ 8112.043586] Key type .llcrypt unregistered [ 8112.045224] Key type ._llcrypt unregistered [ 8119.853915] Key type ._llcrypt registered [ 8119.854979] Key type .llcrypt registered [ 8119.894467] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8126.513619] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8126.822576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8126.846150] alg: No test for adler32 (adler32-zlib) [ 8127.693546] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 8127.768897] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8129.352078] Key type lgssc registered [ 8129.781129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8133.117280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8135.534050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8137.509779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8139.544224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8141.746192] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:33:56 (1763357636) [ 8147.439308] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8148.328568] Key type lgssc unregistered [ 8148.447459] LNet: 479066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8148.449623] LNetError: 479066:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8148.457301] LNet: Removed LNI 192.168.201.116@tcp [ 8148.744369] Key type .llcrypt unregistered [ 8148.745283] Key type ._llcrypt unregistered [ 8155.941126] Key type ._llcrypt registered [ 8155.942187] Key type .llcrypt registered [ 8155.986835] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8156.564883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8156.571898] alg: No test for adler32 (adler32-zlib) [ 8157.467803] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 8157.575421] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8159.176212] Key type lgssc registered [ 8159.746206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.733875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8163.763146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8163.771476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.899972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8164.914202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8164.963102] Lustre: lustre-MDT0000: new disk, initializing [ 8164.997589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.005353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8166.532521] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8171.071522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8171.093771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8171.134078] Lustre: 480239:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8171.162626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8171.166664] Lustre: Skipped 1 previous similar message [ 8171.207065] Lustre: lustre-MDT0001: new disk, initializing [ 8171.248716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8171.260684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8171.264370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8172.576327] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8175.855668] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8177.185488] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8179.359218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8179.384786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8179.470718] Lustre: lustre-OST0000: new disk, initializing [ 8179.472514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8179.500242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.604399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8180.606894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8180.644510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8181.283304] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8184.309362] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8185.551944] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing lnet_if_list [ 8187.161771] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8187.172282] LNet: Added LNI 192.168.201.116@tcp10 [8/256/0/180] [ 8188.893924] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8188.908170] LNet: Added LNI 192.168.201.116@tcp11 [8/256/0/180] [ 8190.139729] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8190.149315] LNet: Added LNI 192.168.201.116@tcp12 [8/256/0/180] [ 8191.427130] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8191.436359] LNet: Added LNI 192.168.201.116@tcp13 [8/256/0/180] [ 8192.758080] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8192.767753] LNet: Added LNI 192.168.201.116@tcp14 [8/256/0/180] [ 8194.146641] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8194.156131] LNet: Added LNI 192.168.201.116@tcp15 [8/256/0/180] [ 8195.577453] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8195.587598] LNet: Added LNI 192.168.201.116@tcp16 [8/256/0/180] [ 8196.981735] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8196.991262] LNet: Added LNI 192.168.201.116@tcp17 [8/256/0/180] [ 8198.451572] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8198.464101] LNet: Added LNI 192.168.201.116@tcp18 [8/256/0/180] [ 8199.979419] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8199.988525] LNet: Added LNI 192.168.201.116@tcp19 [8/256/0/180] [ 8201.383442] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8201.394133] LNet: Added LNI 192.168.201.116@tcp20 [8/256/0/180] [ 8202.791340] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8202.799553] LNet: Added LNI 192.168.201.116@tcp21 [8/256/0/180] [ 8204.252506] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8204.261278] LNet: Added LNI 192.168.201.116@tcp22 [8/256/0/180] [ 8205.736533] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8205.745169] LNet: Added LNI 192.168.201.116@tcp23 [8/256/0/180] [ 8207.229250] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8207.237835] LNet: Added LNI 192.168.201.116@tcp24 [8/256/0/180] [ 8208.875814] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8208.887209] LNet: Added LNI 192.168.201.116@tcp25 [8/256/0/180] [ 8210.490813] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8210.499948] LNet: Added LNI 192.168.201.116@tcp26 [8/256/0/180] [ 8212.162166] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8212.171053] LNet: Added LNI 192.168.201.116@tcp27 [8/256/0/180] [ 8212.178095] LNetError: 479501:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 8212.180739] LNetError: Protocol error connecting to 192.168.201.116@tcp26 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 8213.930924] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8213.941802] LNet: Added LNI 192.168.201.116@tcp28 [8/256/0/180] [ 8213.946674] LNetError: 479500:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 8213.949836] LNetError: Protocol error connecting to 192.168.201.116@tcp26 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 8215.693276] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8215.704057] LNet: Added LNI 192.168.201.116@tcp29 [8/256/0/180] [ 8215.708293] LNetError: 479499:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 8215.713527] LNetError: Protocol error connecting to 192.168.201.116@tcp28 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 8215.968096] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp28 [ 8217.629046] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8217.641101] LNet: Added LNI 192.168.201.116@tcp30 [8/256/0/180] [ 8218.017894] LNetError: 479498:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 8218.023550] LNetError: 479498:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8218.026993] LNetError: Protocol error connecting to 192.168.201.116@tcp28 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 8218.031512] LNetError: Skipped 1 previous similar message [ 8219.626487] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8219.637490] LNet: Added LNI 192.168.201.116@tcp31 [8/256/0/180] [ 8221.557689] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8221.568114] LNet: Added LNI 192.168.201.116@tcp32 [8/256/0/180] [ 8222.117112] LNetError: 479498:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 8222.120291] LNetError: 479498:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 8222.122542] LNetError: Protocol error connecting to 192.168.201.116@tcp29 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 8222.125765] LNetError: Skipped 6 previous similar messages [ 8223.543144] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8223.551505] LNet: Added LNI 192.168.201.116@tcp33 [8/256/0/180] [ 8225.337773] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8225.347734] LNet: Added LNI 192.168.201.116@tcp34 [8/256/0/180] [ 8227.183164] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8227.192786] LNet: Added LNI 192.168.201.116@tcp35 [8/256/0/180] [ 8229.192685] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8229.201608] LNet: Added LNI 192.168.201.116@tcp36 [8/256/0/180] [ 8230.305823] LNetError: 492381:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 8230.310577] LNetError: 492381:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 8230.312645] LNetError: Protocol error connecting to 192.168.201.116@tcp34 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 8230.315354] LNetError: Skipped 7 previous similar messages [ 8231.399252] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8231.416357] LNet: Added LNI 192.168.201.116@tcp37 [8/256/0/180] [ 8233.500833] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8233.510568] LNet: Added LNI 192.168.201.116@tcp38 [8/256/0/180] [ 8235.622411] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8235.631894] LNet: Added LNI 192.168.201.116@tcp39 [8/256/0/180] [ 8237.663891] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8237.676359] LNet: Added LNI 192.168.201.116@tcp40 [8/256/0/180] [ 8238.573306] LNetError: 479508:0:(peer.c:1877:lnet_peer_add_nid()) Peer 192.168.201.116@tcp NID 192.168.201.116@tcp is already locked with peer 192.168.201.116@tcp [ 8238.583167] LNetError: 479508:0:(peer.c:277:lnet_destroy_peer_locked()) ASSERTION( list_empty(&lp->lp_peer_list) ) failed: [ 8238.587024] LNetError: 479508:0:(peer.c:277:lnet_destroy_peer_locked()) LBUG [ 8238.588842] CPU: 2 PID: 479508 Comm: lnet_discovery Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 8238.592086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 8238.594174] Call Trace: [ 8238.595894] ? dump_stack+0xbb/0x10e [ 8238.597546] ? lbug_with_loc.cold.4+0xd/0x86 [libcfs] [ 8238.598735] ? lnet_destroy_peer_locked+0x3cb/0x420 [lnet] [ 8238.599892] ? lnet_peer_discovery_complete+0x4df/0x500 [lnet] [ 8238.601097] ? lnet_peer_discovery+0x7b9/0x1c70 [lnet] [ 8238.602188] ? lnet_discovery_event_reply+0xf60/0xf60 [lnet] [ 8238.603317] ? lnet_peer_data_present+0xca0/0xca0 [lnet] [ 8238.605191] ? kthread+0x1d1/0x200 [ 8238.606726] ? set_kthread_struct+0x70/0x70 [ 8238.608434] ? ret_from_fork+0x1f/0x30 [ 8238.610042] Kernel panic - not syncing: LBUG [ 8238.611882] CPU: 2 PID: 479508 Comm: lnet_discovery Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 8238.616400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 8238.619957] Call Trace: [ 8238.620663] ? dump_stack+0xbb/0x10e [ 8238.621585] ? panic+0x142/0x44c [ 8238.622857] ? lbug_with_loc.cold.4+0x32/0x86 [libcfs] [ 8238.625249] ? lnet_destroy_peer_locked+0x3cb/0x420 [lnet] [ 8238.627688] ? lnet_peer_discovery_complete+0x4df/0x500 [lnet] [ 8238.630081] ? lnet_peer_discovery+0x7b9/0x1c70 [lnet] [ 8238.632220] ? lnet_discovery_event_reply+0xf60/0xf60 [lnet] [ 8238.634467] ? lnet_peer_data_present+0xca0/0xca0 [lnet] [ 8238.636728] ? kthread+0x1d1/0x200 [ 8238.637847] ? set_kthread_struct+0x70/0x70 [ 8238.639442] ? ret_from_fork+0x1f/0x30 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-957.1.3.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 29 14:49:43 UTC 2018 [ 0.000000] Command line: rd.shell ro panic=1 nomodeset ipmtu=9000 rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr disable_cpu_apicid=0 acpi_rsdp=0xf5200 elfcorehdr=3079540K [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ac000000-0x00000000bbf5cfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bbfffc00-0x00000000bbffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] extended physical RAM map: [ 0.000000] reserve setup_data: [mem 0x0000000000000000-0x0000000000000fff] reserved [ 0.000000] reserve setup_data: [mem 0x0000000000001000-0x000000000009fbff] usable [ 0.000000] reserve setup_data: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000ac000000-0x00000000ac00002f] usable [ 0.000000] reserve setup_data: [mem 0x00000000ac000030-0x00000000bbf5cfff] usable [ 0.000000] reserve setup_data: [mem 0x00000000bbfffc00-0x00000000bbffffff] usable [ 0.000000] reserve setup_data: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] reserve setup_data: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: last_pfn = 0xbc000 max_arch_pfn = 0x400000000 [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] x2apic enabled by BIOS, switching to x2apic ops [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] mapped at [ffffffffff2003f0] [ 0.000000] Using GB pages for direct mapping [ 0.000000] RAMDISK: [mem 0xb77a1000-0xba7fffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5200 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1d87 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1c23 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1c97 00090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1d27 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1d5f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] Setting APIC routing to cluster x2apic. [ 0.000000] NUMA turned off [ 0.000000] Faking a node at [mem 0x0000000000000000-0x00000000bbffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0xbbf36000-0xbbf5cfff] [ 0.000000] kvm-clock: cpu 0, msr 0:bbee5001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 8240780520610 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0xac000000-0xbbf5cfff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0xbbf5cfff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: Disabling requested cpu. Processor 0/0x0 ignored. [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: NR_CPUS/possible_cpus limit of 1 almost reached. Keeping one slot for boot cpu. Processor 1/0x1 ignored. [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: NR_CPUS/possible_cpus limit of 1 reached. Processor 3/0x3 ignored. [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: 4 Processors exceeds NR_CPUS limit of 1 [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x00100000-0xabffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xac000000-0xac000fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbbf5d000-0xbbffffff] [ 0.000000] e820: [mem 0x00100000-0xabffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 38 pages/cpu @ffff8800bbc00000 s118784 r8192 d28672 u2097152 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr bbc1c040 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 64485 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: rd.shell ro panic=1 nomodeset ipmtu=9000 rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr disable_cpu_apicid=0 acpi_rsdp=0xf5200 elfcorehdr=3079540K [ 0.000000] audit: disabled (until reboot) [ 0.000000] Misrouted IRQ fixup and polling support enabled [ 0.000000] This may significantly impact system performance [ 0.000000] Disabling memory control group subsystem [ 0.000000] PID hash table entries: 1024 (order: 1, 8192 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 184636k/3080192k available (7664k kernel code, 2818068k absent, 77488k reserved, 6055k data, 1876k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1. [ 0.000000] NR_IRQS:327936 nr_irqs:256 0 [ 0.000000] do_IRQ: 0.236 No irq handler for vector (irq -1) [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.152170] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.153840] pid_max: default: 32768 minimum: 301 [ 0.154813] Security Framework initialized [ 0.155866] SELinux: Initializing. [ 0.156697] Yama: becoming mindful. [ 0.157698] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes) [ 0.159130] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes) [ 0.160773] Mount-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.162374] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.163995] Initializing cgroup subsys memory [ 0.165286] Initializing cgroup subsys devices [ 0.166190] Initializing cgroup subsys freezer [ 0.167052] Initializing cgroup subsys net_cls [ 0.167913] Initializing cgroup subsys blkio [ 0.168868] Initializing cgroup subsys perf_event [ 0.169884] Initializing cgroup subsys hugetlb [ 0.170741] Initializing cgroup subsys pids [ 0.171570] Initializing cgroup subsys net_prio [ 0.172705] x86/cpu: Activated the Intel User Mode Instruction Prevention (UMIP) CPU feature [ 0.175387] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.176445] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.177582] tlb_flushall_shift: 6 [ 0.178312] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.180315] FEATURE SPEC_CTRL Present [ 0.181030] FEATURE IBPB_SUPPORT Present [ 0.182202] Spectre V2 : Mitigation: Full retpoline [ 0.208942] Freeing SMP alternatives: 28k freed [ 0.218250] ACPI: Core revision 20130517 [ 0.219896] ACPI: All ACPI Tables successfully acquired [ 0.221081] ftrace: allocating 29188 entries in 115 pages [ 0.276575] Switched APIC routing to physical x2apic. [ 0.278852] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.280062] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.282044] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.283654] ... version: 2 [ 0.284376] ... bit width: 48 [ 0.285099] ... generic registers: 4 [ 0.285816] ... value mask: 0000ffffffffffff [ 0.287169] ... max period: 00007fffffffffff [ 0.288205] ... fixed-purpose events: 3 [ 0.288947] ... event mask: 000000070000000f [ 0.289965] KVM setup paravirtual spinlock [ 0.292194] Brought up 1 CPUs [ 0.292808] smpboot: Max logical packages: 1 [ 0.293580] smpboot: Total of 1 processors activated (4799.99 BogoMIPS) [ 0.295196] devtmpfs: initialized [ 0.298193] EVM: security.selinux [ 0.298785] EVM: security.ima [ 0.299312] EVM: security.capability [ 0.301454] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.302681] pinctrl core: initialized pinctrl subsystem [ 0.303629] RTC time: 5:35:35, date: 11/17/25 [ 0.304496] NET: Registered protocol family 16 [ 0.305725] ACPI: bus type PCI registered [ 0.306388] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.307665] PCI: Using configuration type 1 for base access [ 0.308720] core: PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off [ 0.311199] ACPI: Added _OSI(Module Device) [ 0.311995] ACPI: Added _OSI(Processor Device) [ 0.312737] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.313569] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314504] ACPI: Added _OSI(Linux-Dell-Video) [ 0.316413] ACPI: Interpreter enabled [ 0.317084] ACPI: (supports S0 S3 S4 S5) [ 0.317824] ACPI: Using IOAPIC for interrupt routing [ 0.318743] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320285] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.324494] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325688] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.326852] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.328009] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.330291] acpiphp: Slot [2] registered [ 0.331079] acpiphp: Slot [5] registered [ 0.331793] acpiphp: Slot [6] registered [ 0.332600] acpiphp: Slot [7] registered [ 0.333371] acpiphp: Slot [8] registered [ 0.334105] acpiphp: Slot [9] registered [ 0.334819] acpiphp: Slot [10] registered [ 0.335528] acpiphp: Slot [3] registered [ 0.336262] acpiphp: Slot [4] registered [ 0.336977] acpiphp: Slot [11] registered [ 0.337730] acpiphp: Slot [12] registered [ 0.338509] acpiphp: Slot [13] registered [ 0.339244] acpiphp: Slot [14] registered [ 0.340028] acpiphp: Slot [15] registered [ 0.340919] acpiphp: Slot [16] registered [ 0.341629] acpiphp: Slot [17] registered [ 0.342366] acpiphp: Slot [18] registered [ 0.343103] acpiphp: Slot [19] registered [ 0.343818] acpiphp: Slot [20] registered [ 0.344553] acpiphp: Slot [21] registered [ 0.345337] acpiphp: Slot [22] registered [ 0.346079] acpiphp: Slot [23] registered [ 0.346817] acpiphp: Slot [24] registered [ 0.347600] acpiphp: Slot [25] registered [ 0.348345] acpiphp: Slot [26] registered [ 0.349121] acpiphp: Slot [27] registered [ 0.349848] acpiphp: Slot [28] registered [ 0.350628] acpiphp: Slot [29] registered [ 0.351372] acpiphp: Slot [30] registered [ 0.352121] acpiphp: Slot [31] registered [ 0.352874] PCI host bridge to bus 0000:00 [ 0.353589] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.354709] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.355878] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357169] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.358467] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.359895] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.368266] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369492] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370555] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371820] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.374320] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375634] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.488030] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 10 *11) [ 0.489418] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.490630] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.491759] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 *10 11) [ 0.492902] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.494682] ACPI: NR_CPUS/possible_cpus limit of 1 reached. Processor 4/0x1 ignored. [ 0.496083] ACPI: Unable to map lapic to logical cpu number [ 0.497118] acpi LNXCPU:02: BIOS reported wrong ACPI id 0 for the processor [ 0.498430] ACPI: NR_CPUS/possible_cpus limit of 1 reached. Processor 5/0x3 ignored. [ 0.499804] ACPI: Unable to map lapic to logical cpu number [ 0.501085] vgaarb: loaded [ 0.501690] SCSI subsystem initialized [ 0.502413] ACPI: bus type USB registered [ 0.503117] usbcore: registered new interface driver usbfs [ 0.504123] usbcore: registered new interface driver hub [ 0.505039] usbcore: registered new device driver usb [ 0.506024] EDAC MC: Ver: 3.0.0 [ 0.506868] PCI: Using ACPI for IRQ routing [ 0.508039] NetLabel: Initializing [ 0.508684] NetLabel: domain hash size = 128 [ 0.509444] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.510339] NetLabel: unlabeled traffic allowed by default [ 0.511458] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.512393] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.516600] amd_nb: Cannot enumerate AMD northbridges [ 0.517528] Switched to clocksource kvm-clock [ 0.524400] pnp: PnP ACPI init [ 0.524984] ACPI: bus type PNP registered [ 0.526187] pnp: PnP ACPI: found 6 devices [ 0.526932] ACPI: bus type PNP unregistered [ 0.534429] NET: Registered protocol family 2 [ 0.535508] TCP established hash table entries: 2048 (order: 2, 16384 bytes) [ 0.536828] TCP bind hash table entries: 2048 (order: 3, 32768 bytes) [ 0.537925] TCP: Hash tables configured (established 2048 bind 2048) [ 0.539160] TCP: reno registered [ 0.539764] UDP hash table entries: 256 (order: 1, 8192 bytes) [ 0.540748] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes) [ 0.541986] NET: Registered protocol family 1 [ 0.542808] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.543866] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.545548] Unpacking initramfs... [ 0.586744] Freeing initrd memory: 49532k freed [ 0.602597] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 0.604251] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 0.605401] RAPL PMU: hw unit of domain package 2^-0 Joules [ 0.606382] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 0.607856] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.609174] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.610603] futex hash table entries: 256 (order: 2, 16384 bytes) [ 0.611884] Initialise system trusted keyring [ 0.641154] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 0.642454] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.645794] zpool: loaded [ 0.646337] zbud: loaded [ 0.647148] VFS: Disk quotas dquot_6.5.2 [ 0.648050] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.649480] msgmni has been set to 457 [ 0.650256] Key type big_key registered [ 0.651645] NET: Registered protocol family 38 [ 0.652516] Key type asymmetric registered [ 0.653329] Asymmetric key parser 'x509' registered [ 0.654252] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 0.655759] io scheduler noop registered [ 0.656674] io scheduler deadline registered (default) [ 0.657655] io scheduler cfq registered [ 0.658434] io scheduler mq-deadline registered [ 0.659423] io scheduler kyber registered [ 0.660284] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.661425] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.662689] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 0.664063] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.665458] ACPI: Power Button [PWRF] [ 0.666260] GHES: HEST is not enabled! [ 0.667146] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 0.692922] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 0.718915] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 0.720410] Non-volatile memory driver v1.3 [ 0.721467] Linux agpgart interface v0.103 [ 0.722381] crash memory driver: version 1.1 [ 0.723238] rdac: device handler registered [ 0.724033] hp_sw: device handler registered [ 0.724805] emc: device handler registered [ 0.725560] alua: device handler registered [ 0.726382] libphy: Fixed MDIO Bus: probed [ 0.727169] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 0.728409] ehci-pci: EHCI PCI platform driver [ 0.729313] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 0.730414] ohci-pci: OHCI PCI platform driver [ 0.731276] uhci_hcd: USB Universal Host Controller Interface driver [ 0.732934] usbcore: registered new interface driver usbserial_generic [ 0.734460] usbserial: USB Serial support registered for generic [ 0.735672] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.738176] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.739065] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.740072] mousedev: PS/2 mouse device common for all mice [ 0.741473] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 0.743361] rtc_cmos 00:05: RTC can wake from S4 [ 0.744743] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 0.746486] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 0.747688] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 0.749233] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 0.750824] cpuidle: using governor menu [ 0.751985] hidraw: raw HID events driver (C) Jiri Kosina [ 0.753042] usbcore: registered new interface driver usbhid [ 0.753958] usbhid: USB HID core driver [ 0.754654] drop_monitor: Initializing network drop monitor service [ 0.755763] TCP: cubic registered [ 0.756395] Initializing XFRM netlink socket [ 0.757261] NET: Registered protocol family 10 [ 0.758192] NET: Registered protocol family 17 [ 0.758924] mpls_gso: MPLS GSO support [ 0.759595] mce: Unable to init device /dev/mcelog (rc: -5) [ 0.760838] Loading compiled-in X.509 certificates [ 0.761689] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 0.763389] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 0.765604] Loaded X.509 cert 'CentOS Linux kernel signing key: e7cef3613a9b8bd012fae7498272159bb1879c65' [ 0.767315] registered taskstats version 1 [ 0.771721] Key type trusted registered [ 0.775418] Key type encrypted registered [ 0.776259] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 0.778918] Magic number: 1:865:567 [ 0.779609] misc network_latency: hash matches [ 0.780478] rtc_cmos 00:05: setting system clock to 2025-11-17 05:35:35 UTC (1763357735) [ 0.783016] Freeing unused kernel memory: 1876k freed [ 0.784403] Write protecting the kernel read-only data: 12288k [ 0.786710] Freeing unused kernel memory: 516k freed [ 0.789057] Freeing unused kernel memory: 600k freed [ 0.810915] loop: module loaded [ 0.817496] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 0.831833] SQUASHFS error: Filesystem uses "unknown" compression. This is not supported mount: /squash/root: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. [ 0.838997] TECH PREVIEW: Overlay filesystem may not be fully supported. [ 0.838997] Please review provided documentation for limitations. [ 0.841373] overlayfs: upper fs does not support xattr. mount: /newroot/squash: mount point does not exist. switch_root: cannot access /init: No such file or directory switch_root: failed to execute /init: No such file or directory [ 0.848477] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 [ 0.848477] [ 0.850109] CPU: 0 PID: 1 Comm: switch_root Tainted: G ------------ T 3.10.0-957.1.3.el7.x86_64 #1 [ 0.852028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.853746] Call Trace: [ 0.854204] [] dump_stack+0x19/0x1b [ 0.855212] [] panic+0xe8/0x21f [ 0.856026] [] do_exit+0xa36/0xa40 [ 0.856931] [] do_group_exit+0x3f/0xa0 [ 0.857879] [] SyS_exit_group+0x14/0x20 [ 0.858775] [] system_call_fastpath+0x22/0x27 [ 0.859965] Kernel Offset: disabled [ 0.860671] Rebooting in 10 seconds..