[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482598141 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002016] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006014] pid_max: default: 32768 minimum: 301 [ 0.007133] LSM: Security Framework initializing [ 0.008049] Yama: becoming mindful. [ 0.009032] SELinux: Initializing. [ 0.010072] *** VALIDATE selinux *** [ 0.018786] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022716] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026018] *** VALIDATE tmpfs *** [ 0.027429] *** VALIDATE proc *** [ 0.029103] *** VALIDATE cgroup *** [ 0.030011] *** VALIDATE cgroup2 *** [ 0.031254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034030] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.038218] debug: unmapping init [mem 0xffffffff87e59000-0xffffffff87e60fff] [ 0.040182] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041764] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043015] ... bit width: 48 [ 0.044013] ... generic registers: 4 [ 0.045015] ... value mask: 0000ffffffffffff [ 0.046014] ... max period: 00007fffffffffff [ 0.047015] ... fixed-purpose events: 3 [ 0.048014] ... event mask: 000000070000000f [ 0.049312] rcu: Hierarchical SRCU implementation. [ 0.051430] smp: Bringing up secondary CPUs ... [ 0.052604] x86: Booting SMP configuration: [ 0.053023] .... node #0, CPUs: #1 #2 #3 [ 0.056422] smp: Brought up 1 node, 4 CPUs [ 0.058016] smpboot: Max logical packages: 1 [ 0.059021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.193407] node 0 deferred pages initialised in 133ms [ 0.197178] devtmpfs: initialized [ 0.198254] x86/mm: Memory block size: 128MB [ 0.200793] gcov: version magic: 0x41383552 [ 0.202157] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.203098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.204272] pinctrl core: initialized pinctrl subsystem [ 0.205221] [ 0.205617] ************************************************************* [ 0.206014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207015] ** ** [ 0.208014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.209013] ** ** [ 0.210016] ** This means that this kernel is built to expose internal ** [ 0.211018] ** IOMMU data structures, which may compromise security on ** [ 0.212014] ** your system. ** [ 0.213020] ** ** [ 0.214019] ** If you see this message and you are not debugging the ** [ 0.215015] ** kernel, report this immediately to your vendor! ** [ 0.216014] ** ** [ 0.217027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.218018] ************************************************************* [ 0.219644] NET: Registered protocol family 16 [ 0.220450] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.221073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.222077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.223427] cpuidle: using governor menu [ 0.226772] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.229574] PCI: Using configuration type 1 for base access [ 0.231123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.240063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.242020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.244200] cryptd: max_cpu_qlen set to 1000 [ 0.246246] ACPI: Added _OSI(Module Device) [ 0.248012] ACPI: Added _OSI(Processor Device) [ 0.249011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255415] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262065] ACPI: Interpreter enabled [ 0.263052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264017] ACPI: Using IOAPIC for interrupt routing [ 0.266108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270254] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280590] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.282046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.287082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.292827] acpiphp: Slot [2] registered [ 0.294205] acpiphp: Slot [5] registered [ 0.295184] acpiphp: Slot [6] registered [ 0.297208] acpiphp: Slot [7] registered [ 0.298179] acpiphp: Slot [8] registered [ 0.300176] acpiphp: Slot [9] registered [ 0.301199] acpiphp: Slot [10] registered [ 0.303210] acpiphp: Slot [3] registered [ 0.305142] acpiphp: Slot [4] registered [ 0.306200] acpiphp: Slot [11] registered [ 0.308178] acpiphp: Slot [12] registered [ 0.309154] acpiphp: Slot [13] registered [ 0.311175] acpiphp: Slot [14] registered [ 0.312146] acpiphp: Slot [15] registered [ 0.314151] acpiphp: Slot [16] registered [ 0.315176] acpiphp: Slot [17] registered [ 0.317166] acpiphp: Slot [18] registered [ 0.318147] acpiphp: Slot [19] registered [ 0.320153] acpiphp: Slot [20] registered [ 0.321149] acpiphp: Slot [21] registered [ 0.323161] acpiphp: Slot [22] registered [ 0.324173] acpiphp: Slot [23] registered [ 0.326195] acpiphp: Slot [24] registered [ 0.327149] acpiphp: Slot [25] registered [ 0.329164] acpiphp: Slot [26] registered [ 0.330185] acpiphp: Slot [27] registered [ 0.332162] acpiphp: Slot [28] registered [ 0.334167] acpiphp: Slot [29] registered [ 0.335177] acpiphp: Slot [30] registered [ 0.337146] acpiphp: Slot [31] registered [ 0.338120] PCI host bridge to bus 0000:00 [ 0.340047] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345072] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.350028] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.352027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357071] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360339] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369702] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.374068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.380022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.385821] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388841] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394971] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.400021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.413027] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.418027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.425784] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.432020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.454021] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.464657] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.471027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.478036] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493050] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.505080] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.517021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.533034] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.543738] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.550031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.558023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.573032] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.583807] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.599027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.614027] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.625149] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.631035] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639035] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.652025] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.663351] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.667434] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.669358] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.672382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.674266] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.679147] iommu: Default domain type: Passthrough [ 0.681523] SCSI subsystem initialized [ 0.683149] ACPI: bus type USB registered [ 0.684106] usbcore: registered new interface driver usbfs [ 0.686095] usbcore: registered new interface driver hub [ 0.688070] usbcore: registered new device driver usb [ 0.689181] pps_core: LinuxPPS API ver. 1 registered [ 0.691013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694066] PTP clock support registered [ 0.696127] EDAC MC: Ver: 3.0.0 [ 0.698154] PCI: Using ACPI for IRQ routing [ 0.699874] NetLabel: Initializing [ 0.701010] NetLabel: domain hash size = 128 [ 0.702008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704078] NetLabel: unlabeled traffic allowed by default [ 0.706143] vgaarb: loaded [ 0.708186] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.709008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.715485] clocksource: Switched to clocksource kvm-clock [ 0.822301] VFS: Disk quotas dquot_6.6.0 [ 0.823798] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.826023] *** VALIDATE ramfs *** [ 0.827230] *** VALIDATE hugetlbfs *** [ 0.829638] pnp: PnP ACPI init [ 0.831968] pnp: PnP ACPI: found 6 devices [ 0.849198] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.852655] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.854952] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.856727] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.858479] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.860142] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.862810] NET: Registered protocol family 2 [ 0.865101] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.869574] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.873271] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.880116] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.884285] TCP: Hash tables configured (established 65536 bind 65536) [ 0.888428] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.891732] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894890] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.897973] NET: Registered protocol family 1 [ 0.900329] RPC: Registered named UNIX socket transport module. [ 0.902926] RPC: Registered udp transport module. [ 0.904546] RPC: Registered tcp transport module. [ 0.906424] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.909056] NET: Registered protocol family 44 [ 0.911037] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.913238] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.915116] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.917452] PCI: CLS 0 bytes, default 64 [ 0.919203] Unpacking initramfs... [ 2.353333] debug: unmapping init [mem 0xffff912a3cc54000-0xffff912a3ffbffff] [ 2.357213] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.359233] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.361811] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.882339] Initialise system trusted keyrings [ 2.885403] Key type blacklist registered [ 2.887326] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.896667] zbud: loaded [ 2.899922] *** VALIDATE nfs *** [ 2.901458] *** VALIDATE nfs4 *** [ 2.903230] pstore: using deflate compression [ 2.906550] Platform Keyring initialized [ 3.006864] NET: Registered protocol family 38 [ 3.010640] Key type asymmetric registered [ 3.011966] Asymmetric key parser 'x509' registered [ 3.013815] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.016975] io scheduler mq-deadline registered [ 3.018834] io scheduler kyber registered [ 3.020553] io scheduler bfq registered [ 3.022349] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.025554] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.028502] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.031260] ACPI: Power Button [PWRF] [ 3.123612] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.215950] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.403135] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.496500] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.689281] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.718490] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.750252] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.775089] Non-volatile memory driver v1.3 [ 3.777107] Linux agpgart interface v0.103 [ 3.819151] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.823426] vda: detected capacity change from 0 to 68530176 [ 3.840881] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.843407] vdb: detected capacity change from 0 to 1073741824 [ 3.860190] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.863249] vdc: detected capacity change from 0 to 2621440000 [ 3.879745] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.882886] vdd: detected capacity change from 0 to 2621440000 [ 3.898116] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.901420] vde: detected capacity change from 0 to 4294967296 [ 3.916216] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.919060] vdf: detected capacity change from 0 to 4294967296 [ 3.925508] libphy: Fixed MDIO Bus: probed [ 3.931874] usbcore: registered new interface driver usbserial_generic [ 3.934394] usbserial: USB Serial support registered for generic [ 3.936938] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.941054] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.942591] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.945190] mousedev: PS/2 mouse device common for all mice [ 3.948666] rtc_cmos 00:05: RTC can wake from S4 [ 3.952821] rtc_cmos 00:05: registered as rtc0 [ 3.954307] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.955483] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.967763] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.969821] intel_pstate: CPU model not supported [ 3.974517] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.979168] hid: raw HID events driver (C) Jiri Kosina [ 3.981063] usbcore: registered new interface driver usbhid [ 3.983466] usbhid: USB HID core driver [ 3.985335] drop_monitor: Initializing network drop monitor service [ 3.988151] Initializing XFRM netlink socket [ 3.990296] NET: Registered protocol family 10 [ 3.993338] Segment Routing with IPv6 [ 3.994959] NET: Registered protocol family 17 [ 3.997231] mpls_gso: MPLS GSO support [ 4.002842] RAS: Correctable Errors collector initialized. [ 4.004800] AVX version of gcm_enc/dec engaged. [ 4.006579] AES CTR mode by8 optimization enabled [ 4.086860] sched_clock: Marking stable (4086829057, 0)->(5032002976, -945173919) [ 4.090850] registered taskstats version 1 [ 4.093362] Loading compiled-in X.509 certificates [ 4.095201] zswap: loaded using pool lzo/zbud [ 4.124629] Key type big_key registered [ 4.139379] Key type encrypted registered [ 4.141292] ima: No TPM chip found, activating TPM-bypass! [ 4.143821] ima: Allocated hash algorithm: sha1 [ 4.145711] ima: No architecture policies found [ 4.147680] evm: Initialising EVM extended attributes: [ 4.149367] evm: security.selinux [ 4.150522] evm: security.ima [ 4.151684] evm: security.capability [ 4.153142] evm: HMAC attrs: 0x1 [ 4.155508] rtc_cmos 00:05: setting system clock to 2025-11-17 01:29:09 UTC (1763342949) [ 4.161982] debug: unmapping init [mem 0xffffffff88e03000-0xffffffff88ffffff] [ 4.165367] debug: unmapping init [mem 0xffffffff87b82000-0xffffffff87e58fff] [ 4.174095] Write protecting the kernel read-only data: 28672k [ 4.178365] debug: unmapping init [mem 0xffffffff86203000-0xffffffff863fffff] [ 4.181244] debug: unmapping init [mem 0xffffffff86b14000-0xffffffff86bfffff] [ 4.221420] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.230260] systemd[1]: Detected virtualization kvm. [ 4.232474] systemd[1]: Detected architecture x86-64. [ 4.234558] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.265321] systemd[1]: No hostname configured. [ 4.267480] systemd[1]: Set hostname to . [ 4.269861] random: systemd: uninitialized urandom read (16 bytes read) [ 4.272507] systemd[1]: Initializing machine ID from random generator. [ 4.321791] random: ln: uninitialized urandom read (6 bytes read) [ 4.419325] random: systemd: uninitialized urandom read (16 bytes read) [ 4.422393] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.427786] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.433361] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.086765] device-mapper: uevent: version 1.0.3 [ 5.089333] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.767118] virtio_net virtio0 ens2: renamed from eth0 [ 5.803694] random: fast init done [ 5.906629] scsi host0: ata_piix [ 5.954434] scsi host1: ata_piix [ 5.955961] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.958285] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.509795] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.658586] random: crng init done [ 10.660151] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.097440] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.276476] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.540850] SELinux: Disabled at runtime. [ 12.599943] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.607766] systemd[1]: Detected virtualization kvm. [ 12.609487] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.120229] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.123408] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.127924] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.131523] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.134503] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.144449] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.149196] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ 13.341895] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.640290] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.945404] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.968917] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.143090] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.155096] EDAC sbridge: Ver: 1.1.2 [ 15.848297] Key type dns_resolver registered [ 16.156197] NFS: Registering the id_resolver key type [ 16.158440] Key type id_resolver registered [ 16.160695] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg156-server login: [ 44.270601] libcfs: loading out-of-tree module taints kernel. [ 44.286939] Key type ._llcrypt registered [ 44.288311] Key type .llcrypt registered [ 44.340575] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 57.880777] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 59.008062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 59.026612] alg: No test for adler32 (adler32-zlib) [ 60.316985] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 60.755057] hrtimer: interrupt took 6050023 ns [ 61.013150] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 62.744174] Key type lgssc registered [ 64.263319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 78.820670] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.200867] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.887835] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.306721] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.332162] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.901417] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 120.542290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.636616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 120.651161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 121.903971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 121.943818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 122.060733] Lustre: lustre-MDT0000: new disk, initializing [ 122.182327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 122.209310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 126.327262] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.290805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 137.360697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 137.410719] Lustre: 6507:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 137.432805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 137.436208] Lustre: Skipped 1 previous similar message [ 137.497532] Lustre: lustre-MDT0001: new disk, initializing [ 137.551751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 137.571755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 137.584365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 140.158802] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.777228] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 151.043887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 151.122188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 151.315712] Lustre: lustre-OST0000: new disk, initializing [ 151.327365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 151.380777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.163056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 155.173798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 155.217285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 155.997499] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.050288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.092447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 167.151035] Lustre: lustre-OST0001: new disk, initializing [ 167.155521] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 167.204643] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.013849] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.606512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 174.618765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 174.650702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 179.466957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 183.796933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 193.254144] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing check_logdir /tmp/testlogs/ [ 197.229421] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing yml_node [ 201.016420] Lustre: DEBUG MARKER: Client: 2.16.61.47 [ 202.853831] Lustre: DEBUG MARKER: MDS: 2.16.61.47 [ 204.615651] Lustre: DEBUG MARKER: OSS: 2.16.61.47 [ 205.937324] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 20:32:30 EST 2025 [ 222.019621] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 223.954188] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 240.102300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 240.106081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 240.112241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 241.121619] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 241.122392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 241.126669] Lustre: Skipped 2 previous similar messages [ 241.130448] Lustre: Skipped 2 previous similar messages [ 244.937742] LustreError: 12211:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 245.010559] Lustre: server umount lustre-MDT0000 complete [ 248.443899] LustreError: 6500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343193 with bad export cookie 17006713967728350421 [ 248.445217] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 248.452475] LustreError: 6500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 251.364224] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 251.365464] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.366843] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 251.366849] Lustre: Skipped 1 previous similar message [ 251.390872] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 251.392050] Lustre: Skipped 1 previous similar message [ 253.418425] LustreError: 12005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 253.420420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 253.436403] LustreError: 12005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 253.443258] Lustre: Skipped 2 previous similar messages [ 254.628402] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 254.631026] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 254.785280] Lustre: server umount lustre-MDT0001 complete [ 258.354081] LustreError: 12614:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 258.364808] LustreError: 12614:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 258.442210] Lustre: server umount lustre-OST0000 complete [ 261.809894] LustreError: 12814:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 261.813061] LustreError: 12814:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 261.896862] Lustre: server umount lustre-OST0001 complete [ 267.916757] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 275.439497] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 283.057750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 288.155867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 293.925552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 299.286924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 303.947921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 303.990352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 304.138108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 304.168722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 304.231561] Lustre: lustre-MDT0000: new disk, initializing [ 304.293875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 304.308090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 307.062347] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 315.461978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.516990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 315.580504] Lustre: 14998:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 315.645632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 315.649064] Lustre: Skipped 1 previous similar message [ 315.726898] Lustre: lustre-MDT0001: new disk, initializing [ 315.822602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 315.890409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 315.906508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 318.907056] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 324.703265] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 327.637174] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 333.943326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.061117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 334.340102] Lustre: lustre-OST0000: new disk, initializing [ 334.351298] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 334.475688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 336.263725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 336.269894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 336.358786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 338.889629] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 345.690742] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 348.811339] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 348.968358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 352.265400] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 352.474792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 356.841345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 356.841920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.853332] LustreError: Skipped 1 previous similar message [ 356.854434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 356.870849] Lustre: Skipped 1 previous similar message [ 360.248606] LustreError: 17935:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 360.258515] LustreError: 17935:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 360.367191] Lustre: server umount lustre-OST0000 complete [ 367.073478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.076503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 367.076927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.076932] Lustre: Skipped 1 previous similar message [ 367.098337] Lustre: Skipped 1 previous similar message [ 369.384461] LustreError: 18137:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 369.388080] LustreError: 18137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 369.455930] Lustre: server umount lustre-MDT0000 complete [ 372.194923] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.203925] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 372.349641] LustreError: 14990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343317 with bad export cookie 17006713967728351954 [ 372.352768] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.360847] LustreError: 14990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 372.638603] Lustre: server umount lustre-MDT0001 complete [ 376.124377] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:35:20 (1763343320) [ 378.239781] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 379.824770] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:35:24 (1763343324) [ 384.996125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.386145] LustreError: 19012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 385.451601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.737160] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 390.627884] LustreError: 19012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.526279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 399.820468] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 404.481685] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 407.611062] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 412.706400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 417.521493] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 421.826751] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 440.807069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.810478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.817514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 440.817522] Lustre: Skipped 1 previous similar message [ 440.818642] Lustre: Skipped 1 previous similar message [ 444.249291] LustreError: 21278:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 444.258847] LustreError: 21278:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 444.375382] Lustre: server umount lustre-OST0000 complete [ 447.976946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 462.304118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 462.407219] Lustre: server umount lustre-MDT0000 complete [ 465.027072] LustreError: 18992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343410 with bad export cookie 17006713967728352836 [ 465.032305] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.034829] LustreError: 18992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 465.239346] Lustre: server umount lustre-MDT0001 complete [ 473.451427] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 475.706601] Key type lgssc unregistered [ 475.947397] LNet: 22215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.958774] LNetError: 22215:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 475.989972] LNet: Removed LNI 192.168.201.156@tcp [ 476.764159] Key type .llcrypt unregistered [ 476.770076] Key type ._llcrypt unregistered [ 493.455360] Key type ._llcrypt registered [ 493.457097] Key type .llcrypt registered [ 493.539113] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 494.433496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 494.450716] alg: No test for adler32 (adler32-zlib) [ 495.425099] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 495.605527] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 497.264157] Key type lgssc registered [ 498.275488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 505.303375] loop: module loaded [ 505.357226] loop0: detected capacity change from 0 to 204800000 [ 505.554500] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 505.561756] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 505.825031] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 505.882033] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 510.949996] loop0: detected capacity change from 0 to 204800000 [ 511.085804] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 511.102191] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 511.121267] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 511.238290] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 515.340845] loop0: detected capacity change from 0 to 204800000 [ 520.186794] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 520.226534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 520.236192] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.409760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 521.444617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 521.522821] Lustre: lustre-MDT0000: new disk, initializing [ 521.582736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.597780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 524.397655] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 531.433343] loop1: detected capacity change from 0 to 204800000 [ 535.849533] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 535.901450] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 536.215448] Lustre: lustre-OST07c6: new disk, initializing [ 536.220225] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 536.222496] Lustre: Skipped 1 previous similar message [ 536.293087] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 541.226718] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 545.839899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 545.855123] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 545.876382] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 552.002546] Lustre: 24354:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 552.007185] Lustre: 24354:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 552.010619] Lustre: 24354:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 552.014536] Lustre: 24354:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 552.017969] Lustre: 24354:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 552.021267] Lustre: 24354:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 567.542885] Lustre: 25853:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 567.552542] Lustre: 25853:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 567.557074] Lustre: 25853:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 567.563864] Lustre: 25853:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 567.568257] Lustre: 25853:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 567.571389] Lustre: 25853:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 571.984344] Lustre: 24354:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 572.002412] Lustre: 24354:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 572.012722] Lustre: 24354:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 572.017169] Lustre: 24354:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 572.035150] Lustre: 24354:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 572.038101] Lustre: 24354:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 572.040966] Lustre: 24354:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 572.043692] Lustre: 24354:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 572.061323] Lustre: 24354:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 572.068757] Lustre: 24354:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 572.079773] Lustre: 24354:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 572.094563] Lustre: 24354:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 575.259543] Lustre: 24354:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 575.269339] Lustre: 24354:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 575.274102] Lustre: 24354:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 575.283850] Lustre: 24354:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 575.289517] Lustre: 24354:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 575.297052] Lustre: 24354:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 575.308885] Lustre: 24354:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 575.313914] Lustre: 24354:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 575.320044] Lustre: 24354:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 575.326562] Lustre: 24354:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 575.331425] Lustre: 24354:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 575.336021] Lustre: 24354:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.725859] Lustre: 24354:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 579.732230] Lustre: 24354:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 579.737264] Lustre: 24354:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 579.741995] Lustre: 24354:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.746210] Lustre: 24354:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 579.750796] Lustre: 24354:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.755041] Lustre: 24354:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 579.759540] Lustre: 24354:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.769267] Lustre: 24354:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 579.772919] Lustre: 24354:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.803563] Lustre: 24354:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 579.819495] Lustre: 24354:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 590.272393] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 590.294875] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 590.299664] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 590.317945] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 590.321647] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 590.325275] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 590.328975] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 590.332479] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 590.336037] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 590.339100] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 590.342569] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 590.345901] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 637.924728] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.927362] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 637.964818] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 643.041698] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 648.160962] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 650.208138] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 650.248207] LustreError: 26542:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 650.351341] Lustre: server umount lustre-OST07c6 complete [ 657.134642] LustreError: 27045:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 657.138231] LustreError: 27045:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 657.265558] Lustre: server umount lustre-MDT0000 complete [ 668.531412] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 671.418413] Key type lgssc unregistered [ 671.628645] LNet: 27879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.637393] LNetError: 27879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.653700] LNet: Removed LNI 192.168.201.156@tcp [ 672.434957] Key type .llcrypt unregistered [ 672.436968] Key type ._llcrypt unregistered [ 689.305687] Key type ._llcrypt registered [ 689.306894] Key type .llcrypt registered [ 689.397134] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 690.126044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 690.230735] alg: No test for adler32 (adler32-zlib) [ 691.165359] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 691.298714] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 692.928517] Key type lgssc registered [ 693.558920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 698.579522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 698.606670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.991300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.273483] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 710.118567] LustreError: 29133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.125301] LustreError: 29133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 710.860030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 711.203658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 714.972813] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 719.573710] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 723.408500] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 730.751867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 731.215641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.853503] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 743.332279] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 745.526883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 760.805382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.817125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 760.841392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 760.868426] Lustre: Skipped 1 previous similar message [ 765.926496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 765.940417] Lustre: Skipped 1 previous similar message [ 765.981619] LustreError: 31583:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 766.086638] Lustre: server umount lustre-OST0000 complete [ 771.043561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.050110] Lustre: Skipped 1 previous similar message [ 771.053848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.057881] Lustre: Skipped 1 previous similar message [ 774.899712] LustreError: 31784:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 774.907825] LustreError: 31784:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 774.982836] Lustre: server umount lustre-MDT0000 complete [ 776.163552] LustreError: 29846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 776.175030] LustreError: 29846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 777.423037] LustreError: 29115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343722 with bad export cookie 8107444835437228960 [ 777.425478] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 777.432774] LustreError: 29115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 777.559893] LustreError: 31987:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 777.565522] LustreError: 31987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 777.688423] Lustre: server umount lustre-MDT0001 complete [ 786.095730] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 788.566269] Key type lgssc unregistered [ 788.826740] LNet: 32520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.845602] LNetError: 32520:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.873878] LNet: Removed LNI 192.168.201.156@tcp [ 789.677643] Key type .llcrypt unregistered [ 789.681126] Key type ._llcrypt unregistered [ 804.681651] Key type ._llcrypt registered [ 804.683621] Key type .llcrypt registered [ 804.767797] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 805.579522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.597645] alg: No test for adler32 (adler32-zlib) [ 806.755742] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 807.056492] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 808.792195] Key type lgssc registered [ 810.104697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 818.581987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 818.593653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.024259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.953310] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 829.307304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 829.586401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.560505] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 836.333239] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 839.183309] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 845.152801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 845.621636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.686682] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 856.056636] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 865.094884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 879.588769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 879.591898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.605562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.613167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.619255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.623896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.654039] Lustre: Skipped 1 previous similar message [ 882.251467] LustreError: 36017:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 882.370536] Lustre: server umount lustre-OST0000 complete [ 890.850511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.853164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.858483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 890.868572] Lustre: Skipped 1 previous similar message [ 892.267793] LustreError: 36219:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 892.279436] LustreError: 36219:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 892.372558] Lustre: server umount lustre-MDT0000 complete [ 895.326116] LustreError: 36220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343840 with bad export cookie 15662481722034488658 [ 895.330373] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.347574] LustreError: 36220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 895.518655] LustreError: 36421:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 895.532342] LustreError: 36421:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 895.703813] Lustre: server umount lustre-MDT0001 complete [ 904.225691] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 907.023084] Key type lgssc unregistered [ 907.298144] LNet: 36955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.314973] LNetError: 36955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.399145] LNet: Removed LNI 192.168.201.156@tcp [ 909.302083] Key type .llcrypt unregistered [ 909.303904] Key type ._llcrypt unregistered [ 930.607650] Key type ._llcrypt registered [ 930.609364] Key type .llcrypt registered [ 930.711723] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 931.673695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 931.689161] alg: No test for adler32 (adler32-zlib) [ 932.735326] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 932.917856] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 934.576178] Key type lgssc registered [ 935.305737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.782764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.834059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.305443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.518459] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 952.807450] LustreError: 38342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.822300] LustreError: 38342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 954.763055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 955.348363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 959.068449] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 964.061795] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 967.352158] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 973.555404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 973.871873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 979.187246] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 984.102096] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 989.235462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 996.179569] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 996.322971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 996.328224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.600774] LustreError: 40592:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 997.717628] Lustre: server umount lustre-MDT0000 complete [ 999.399553] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.406040] LustreError: 38343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.461064] LustreError: 38343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1001.269806] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.188148] LustreError: 38324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343947 with bad export cookie 4554484399456499832 [ 1002.192339] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.206169] LustreError: 38324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1004.513781] LustreError: 39653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.519554] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.633555] Lustre: Skipped 1 previous similar message [ 1004.636116] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1004.638742] Lustre: Skipped 1 previous similar message [ 1006.420675] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.56@tcp (stopping) [ 1008.389779] LustreError: 40793:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1008.410143] LustreError: 40793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1008.629569] Lustre: server umount lustre-MDT0001 complete [ 1037.280133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1037.319771] LustreError: 41018:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1037.323075] LustreError: 41018:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1037.453773] Lustre: server umount lustre-OST0000 complete [ 1050.564366] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1052.915733] Key type lgssc unregistered [ 1053.230754] LNet: 41857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.237883] LNetError: 41857:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.251804] LNet: Removed LNI 192.168.201.156@tcp [ 1053.996409] Key type .llcrypt unregistered [ 1054.005736] Key type ._llcrypt unregistered [ 1070.798709] Key type ._llcrypt registered [ 1070.800725] Key type .llcrypt registered [ 1070.871790] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1071.495474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1071.563884] alg: No test for adler32 (adler32-zlib) [ 1072.586248] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 1072.750255] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1074.392318] Key type lgssc registered [ 1075.384914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1081.971332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1081.990619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.411694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.034159] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1093.607402] LustreError: 43234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.618319] LustreError: 43234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1095.914674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1096.353616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1101.355320] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1107.299389] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1111.120298] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1118.072985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1118.707518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.740573] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1127.945428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1132.888785] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1140.053725] Lustre: lustre-OST0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 1141.223489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.230636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.247001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.270062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1143.274273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.274635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.091143] LustreError: 45480:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1145.169974] Lustre: server umount lustre-OST0000 complete [ 1162.721289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1162.727550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.742162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.757649] Lustre: Skipped 1 previous similar message [ 1167.612393] LustreError: 45861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1167.620692] LustreError: 45861:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1167.699416] Lustre: server umount lustre-MDT0000 complete [ 1167.843465] LustreError: 43234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.861468] LustreError: 43234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1171.119869] LustreError: 43214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344116 with bad export cookie 12378152586833169104 [ 1171.120312] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.123782] LustreError: 43214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1177.326412] LustreError: 46064:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1177.337350] LustreError: 46064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1177.471176] Lustre: server umount lustre-MDT0001 complete [ 1187.870400] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1190.666437] Key type lgssc unregistered [ 1191.066176] LNet: 46595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.081929] LNetError: 46595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.094820] LNet: Removed LNI 192.168.201.156@tcp [ 1191.834658] Key type .llcrypt unregistered [ 1191.837341] Key type ._llcrypt unregistered [ 1211.325401] Key type ._llcrypt registered [ 1211.327339] Key type .llcrypt registered [ 1211.381925] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1212.018905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.046833] alg: No test for adler32 (adler32-zlib) [ 1213.066645] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 1213.228993] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1214.872447] Key type lgssc registered [ 1215.742457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.451879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.468967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.810461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.544479] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1232.227915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1232.563263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.652586] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1239.546651] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1242.218408] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1247.051835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1247.295676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.309957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1251.045164] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1255.589486] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1263.979693] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1267.147917] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1279.117210] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1620h50m22s [ 1280.626709] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1282.908445] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 1283.044205] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.067681] Lustre: Skipped 1 previous similar message [ 1283.955789] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 1283.974091] Lustre: Skipped 3 previous similar messages [ 1288.304405] LustreError: 50569:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1288.389082] Lustre: server umount lustre-MDT0000 complete [ 1289.013972] LustreError: 47999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.021072] LustreError: 47999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1290.212574] LustreError: 48715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.231709] LustreError: 48715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1291.667855] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1295.181253] LustreError: 47999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.484460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.658261] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.910395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.969400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1298.925701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1301.630663] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1304.827378] Lustre: DEBUG MARKER: Wait for df (36022) ... [ 1306.101534] Lustre: DEBUG MARKER: done [ 1313.249212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1313.258488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.271739] Lustre: Skipped 1 previous similar message [ 1313.275666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1313.278648] Lustre: Skipped 3 previous similar messages [ 1313.767766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1315.254182] LustreError: 51624:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1315.265034] LustreError: 51624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1315.382358] Lustre: server umount lustre-OST0000 complete [ 1319.394743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1319.395559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.417795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.417804] Lustre: Skipped 3 previous similar messages [ 1319.452415] Lustre: Skipped 2 previous similar messages [ 1325.294235] LustreError: 51827:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1325.297044] LustreError: 51827:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1325.391776] Lustre: server umount lustre-MDT0000 complete [ 1325.537800] LustreError: 49213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1325.588163] LustreError: 49213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1328.920147] LustreError: 47982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344274 with bad export cookie 9864359895580397037 [ 1328.927913] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1329.079647] LustreError: 52028:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1329.126160] LustreError: 52028:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1329.392950] Lustre: server umount lustre-MDT0001 complete [ 1343.619179] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1346.892704] Key type lgssc unregistered [ 1347.245117] LNet: 52560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.251791] LNetError: 52560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.273055] LNet: Removed LNI 192.168.201.156@tcp [ 1348.436900] Key type .llcrypt unregistered [ 1348.444239] Key type ._llcrypt unregistered [ 1386.333491] Key type ._llcrypt registered [ 1386.335547] Key type .llcrypt registered [ 1386.466354] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 1402.829709] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1403.863659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.986200] alg: No test for adler32 (adler32-zlib) [ 1405.016814] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 1405.181387] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1406.842682] Key type lgssc registered [ 1407.815600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.987918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1422.643495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1428.473813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1435.213473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1441.874715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1441.928779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1441.956106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.234325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1443.263930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1443.337784] Lustre: lustre-MDT0000: new disk, initializing [ 1443.427069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.444134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1446.867373] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1456.944598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1456.995313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1457.068941] Lustre: 56511:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1457.127648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1457.130790] Lustre: Skipped 1 previous similar message [ 1457.198155] Lustre: lustre-MDT0001: new disk, initializing [ 1457.353897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.385174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1457.394902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1460.827361] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1467.944926] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1471.623284] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1479.002973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1479.097083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1479.415992] Lustre: lustre-OST0000: new disk, initializing [ 1479.419136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1479.539339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.925825] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1488.428582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1488.446342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1488.621917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1492.974245] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1496.776346] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1496.999403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1501.029945] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1501.245862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1504.225526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1504.227864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.234362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.238797] Lustre: Skipped 1 previous similar message [ 1509.314347] LustreError: 59453:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1509.460801] Lustre: server umount lustre-OST0000 complete [ 1513.956560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1513.972282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.010452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1514.035156] Lustre: Skipped 1 previous similar message [ 1519.586415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.598533] Lustre: Skipped 3 previous similar messages [ 1519.957817] LustreError: 59655:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1519.979252] LustreError: 59655:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1520.216236] Lustre: server umount lustre-MDT0000 complete [ 1524.706371] LustreError: 58868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.730931] LustreError: 58868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1525.138618] LustreError: 56505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344470 with bad export cookie 10277065031191860456 [ 1525.140768] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1525.167939] LustreError: 56505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1525.343710] LustreError: 59858:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1525.347301] LustreError: 59858:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1525.508158] Lustre: server umount lustre-MDT0001 complete [ 1534.831251] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 20:54:38 (1763344478) [ 1541.631286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.132793] LustreError: 60531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.214614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1546.169315] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1547.235247] LustreError: 60531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.354848] LustreError: 60531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.275487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1554.534375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1558.340552] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1563.427747] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1566.714572] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1572.898359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1573.157102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.869456] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1582.796894] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1592.891634] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1620h55m37s [ 1594.741494] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1601.232162] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1603.349793] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1603.524388] LustreError: 62687:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1623.908645] Lustre: lustre-MDT0000: Client a811c601-ddc8-47f3-b1d1-5a05dd697c72 (at 192.168.201.56@tcp) reconnecting [ 1643.536493] LustreError: 62687:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1646.301309] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1649.415812] Lustre: DEBUG MARKER: done [ 1656.294067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1656.302862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.321549] Lustre: Skipped 1 previous similar message [ 1656.326476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1659.373893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1659.377917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.490624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.499338] Lustre: Skipped 3 previous similar messages [ 1669.088118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1669.113119] LustreError: 63532:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1669.118384] LustreError: 63532:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1669.213725] Lustre: server umount lustre-OST0000 complete [ 1672.161363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1672.168636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.256230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.262942] Lustre: Skipped 3 previous similar messages [ 1678.184753] LustreError: 63735:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1678.187548] LustreError: 63735:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1678.266483] Lustre: server umount lustre-MDT0000 complete [ 1680.814283] LustreError: 62148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344626 with bad export cookie 10277065031191861338 [ 1680.821757] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.827957] LustreError: 62148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1681.057304] Lustre: server umount lustre-MDT0001 complete [ 1689.189548] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1691.395770] Key type lgssc unregistered [ 1691.672314] LNet: 64468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.679091] LNetError: 64468:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.714625] LNet: Removed LNI 192.168.201.156@tcp [ 1693.413131] Key type .llcrypt unregistered [ 1693.414908] Key type ._llcrypt unregistered [ 1722.580679] Key type ._llcrypt registered [ 1722.582967] Key type .llcrypt registered [ 1722.758460] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 1740.195849] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1741.685625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1741.717188] alg: No test for adler32 (adler32-zlib) [ 1742.869622] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 1743.185477] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1744.888716] Key type lgssc registered [ 1746.157770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1755.433893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1762.679639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1769.730147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1776.991562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1783.659446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1783.716460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1783.731842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.042317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1785.096745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1785.200099] Lustre: lustre-MDT0000: new disk, initializing [ 1785.313241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.336068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1789.090945] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1798.954231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1799.025929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1799.095824] Lustre: 68420:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1799.159478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1799.162439] Lustre: Skipped 1 previous similar message [ 1799.227537] Lustre: lustre-MDT0001: new disk, initializing [ 1799.334017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.407202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1799.416634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1802.781793] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1808.955352] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1812.836797] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1818.779588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1818.851901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1819.092462] Lustre: lustre-OST0000: new disk, initializing [ 1819.099866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1819.176188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.783484] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1825.849486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1825.862755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1825.936200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1830.389777] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1834.261973] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1834.485736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1837.838379] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1838.024430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1841.634150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.636355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.637689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.660890] Lustre: Skipped 1 previous similar message [ 1845.688117] LustreError: 71354:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1845.776441] Lustre: server umount lustre-OST0000 complete [ 1850.850315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1850.854144] LustreError: Skipped 1 previous similar message [ 1850.862497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.879799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.885965] Lustre: Skipped 1 previous similar message [ 1851.877254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.889540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.895434] Lustre: Skipped 1 previous similar message [ 1854.768282] LustreError: 71558:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1854.770638] LustreError: 71558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1854.879936] Lustre: server umount lustre-MDT0000 complete [ 1858.361792] LustreError: 68412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344803 with bad export cookie 18434126389182167100 [ 1858.380921] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.462754] LustreError: 71760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1858.465604] LustreError: 71760:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1858.611671] Lustre: server umount lustre-MDT0001 complete [ 1865.902729] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 21:00:10 (1763344810) [ 1867.216071] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1868.783731] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 21:00:13 (1763344813) [ 1879.269785] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1883.280744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.723986] LustreError: 72810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.742194] LustreError: 72810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1883.776172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.653442] Lustre: Failing over lustre-MDT0000 [ 1884.661666] LustreError: 72838:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1884.670257] LustreError: 72838:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1884.690660] LustreError: 72838:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1884.840769] LustreError: 72889:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1884.843832] LustreError: 72889:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1885.001891] Lustre: server umount lustre-MDT0000 complete [ 1890.513822] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:00:34 (1763344834) [ 1897.021908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.273859] LustreError: 73456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.329861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.793949] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1902.588899] LustreError: 73456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.724183] LustreError: 73456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.312603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1908.585926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1911.629929] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1915.944700] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1919.341620] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1925.405278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1925.626362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.932340] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1934.267556] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1948.347959] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1954.788821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1954.793460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.815314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.969384] LustreError: 75838:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1954.972862] LustreError: 75838:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1955.028387] Lustre: server umount lustre-MDT0000 complete [ 1957.347060] LustreError: 75086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.360769] LustreError: 75086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1957.715267] LustreError: 73437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344903 with bad export cookie 18434126389182168262 [ 1957.717673] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1957.736223] LustreError: 73437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1957.946317] Lustre: server umount lustre-MDT0001 complete [ 1960.432939] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1963.974325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1970.080890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.824617] Lustre: 66632:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344907/real 1763344907] req@0000000053022927 x1849000919635200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763344923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1977.841743] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.850971] Lustre: Skipped 2 previous similar messages [ 1982.944628] Lustre: 66629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344912/real 1763344912] req@0000000084ffef41 x1849000919635328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763344928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.014759] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xffd32c4863db8436 [ 1983.025501] Lustre: MGC192.168.201.156@tcp: Connection restored to 0@lo (at 0@lo) [ 1983.321214] LustreError: 75077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.425923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.539914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1984.355106] Lustre: 66630:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344913/real 1763344913] req@000000008013f40f x1849000919635584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763344929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1986.764573] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1990.631765] Lustre: 66631:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344919/real 1763344919] req@00000000e4c300f3 x1849000919635712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763344935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1993.828948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1997.166824] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1997.523957] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2003.076805] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2007.295327] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2013.649227] LustreError: 78298:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2013.689369] LustreError: 78298:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2013.770146] Lustre: server umount lustre-MDT0000 complete [ 2014.696863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.708922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.743245] LustreError: 75076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.753253] LustreError: 75076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2017.430258] LustreError: 75839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344962 with bad export cookie 18434126389182170166 [ 2017.434947] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.772371] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2019.823838] Lustre: server umount lustre-MDT0001 complete [ 2025.422952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2034.254938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.336759] LustreError: 66628:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000093f775a2 x1849000919659520/t0(0) o250->MGC192.168.201.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2042.643297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2045.362793] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2048.994972] LustreError: 75076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.002338] LustreError: 75076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2052.023869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2052.394930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.400546] Lustre: Skipped 2 previous similar messages [ 2055.264657] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2058.691549] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2058.739394] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2058.744439] Lustre: Skipped 1 previous similar message [ 2061.107891] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2067.940468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2067.951267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.968782] Lustre: Skipped 2 previous similar messages [ 2067.974985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.770059] LustreError: 80756:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2071.773916] LustreError: 80756:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2071.853878] Lustre: server umount lustre-MDT0000 complete [ 2074.873321] LustreError: 73436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345020 with bad export cookie 18434126389182171538 [ 2074.879589] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2075.087534] Lustre: server umount lustre-MDT0001 complete [ 2080.563232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2083.554065] Lustre: DEBUG MARKER: files compared the same [ 2095.656809] Lustre: server umount lustre-OST0000 complete [ 2111.191927] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2113.433919] Key type lgssc unregistered [ 2113.674742] LNet: 82388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.686734] LNetError: 82388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.710032] LNet: Removed LNI 192.168.201.156@tcp [ 2114.417858] Key type .llcrypt unregistered [ 2114.419587] Key type ._llcrypt unregistered [ 2133.566516] Key type ._llcrypt registered [ 2133.572819] Key type .llcrypt registered [ 2133.673432] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2134.284242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2134.416073] alg: No test for adler32 (adler32-zlib) [ 2135.431419] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 2135.624411] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2137.272545] Key type lgssc registered [ 2138.085757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2143.836300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2143.851331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.216700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.763391] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug +malloc all [ 2154.409397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2154.674288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2157.342917] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug +malloc all [ 2161.201224] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2163.632771] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2168.640724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2168.830856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2170.858173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2172.958614] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug +malloc all [ 2176.856060] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2186.209100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2186.213954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.221800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.282133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.285306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2189.292500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.762491] LustreError: 86015:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2191.851453] Lustre: server umount lustre-OST0000 complete [ 2195.937635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2195.941447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.952130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2195.960526] Lustre: Skipped 1 previous similar message [ 2199.526292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.529415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.791610] LustreError: 86219:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2200.797165] LustreError: 86219:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2200.839626] Lustre: server umount lustre-MDT0000 complete [ 2204.105494] LustreError: 83746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345149 with bad export cookie 11039422229198941748 [ 2204.108196] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.112468] LustreError: 83746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2204.224923] LustreError: 86420:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2204.229164] LustreError: 86420:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2204.342511] Lustre: server umount lustre-MDT0001 complete [ 2212.192298] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2214.576552] Key type lgssc unregistered [ 2214.913064] LNet: 86952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.924047] LNetError: 86952:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.947810] LNet: Removed LNI 192.168.201.156@tcp [ 2215.632894] Key type .llcrypt unregistered [ 2215.635444] Key type ._llcrypt unregistered [ 2234.293611] Key type ._llcrypt registered [ 2234.297966] Key type .llcrypt registered [ 2234.386812] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2235.128969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2235.152030] alg: No test for adler32 (adler32-zlib) [ 2236.174273] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 2236.361294] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2238.032266] Key type lgssc registered [ 2239.389691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.141664] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.155216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2262.624823] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2267.745105] LustreError: 88302:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 2294.432668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.063593] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2303.247674] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2309.052188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.176054] Lustre: *** cfs_fail_loc=706, val=0*** [ 2309.484854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.553643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2312.744512] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2314.724812] LustreError: 88328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.755552] LustreError: 88328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2319.849151] LustreError: 88328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.202288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2321.678846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2324.753873] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2329.174732] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2332.067824] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2334.185668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.197897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.307384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.316571] Lustre: Skipped 1 previous similar message [ 2339.583112] LustreError: 90624:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2339.686977] Lustre: server umount lustre-OST0000 complete [ 2344.422126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.425217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2344.449877] Lustre: Skipped 2 previous similar messages [ 2344.456806] Lustre: Skipped 2 previous similar messages [ 2349.537157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.545959] Lustre: Skipped 1 previous similar message [ 2354.657290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2354.661934] Lustre: Skipped 1 previous similar message [ 2357.216229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2357.370675] LustreError: 90826:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2357.379895] LustreError: 90826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2357.533304] Lustre: server umount lustre-MDT0000 complete [ 2359.778293] LustreError: 89274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.817588] LustreError: 89274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2361.322077] LustreError: 88309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345306 with bad export cookie 9746681099713235422 [ 2361.332153] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2361.418475] LustreError: 91029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2361.420644] LustreError: 91029:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2361.544324] Lustre: server umount lustre-MDT0001 complete [ 2371.783625] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2374.457709] Key type lgssc unregistered [ 2374.701270] LNet: 91564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.710568] LNetError: 91564:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.721385] LNet: Removed LNI 192.168.201.156@tcp [ 2375.339282] Key type .llcrypt unregistered [ 2375.342539] Key type ._llcrypt unregistered [ 2394.772553] Key type ._llcrypt registered [ 2394.773683] Key type .llcrypt registered [ 2394.829612] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2395.218340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2395.227033] alg: No test for adler32 (adler32-zlib) [ 2396.162990] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 2396.291801] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2397.896171] Key type lgssc registered [ 2398.605098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2403.880990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2403.903763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.444840] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2414.426423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2414.990570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2417.976632] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2420.193172] LustreError: 93610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.210860] LustreError: 93610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2423.812913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2424.083828] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.111790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.115241] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2425.125207] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.155635] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2428.392538] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2430.439054] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.478809] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2435.589815] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.617744] LustreError: 93614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2436.479876] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.503701] Lustre: MGS: Client 58252ba5-f641-4d97-aec9-9743132bc8d1 (at 0@lo) reconnecting [ 2436.756209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.837374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2440.874385] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2457.058457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2457.063668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.080931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.571378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.585572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.690569] LustreError: 95573:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2460.786357] Lustre: server umount lustre-OST0000 complete [ 2467.815710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.817365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2467.818191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2467.826162] Lustre: Skipped 1 previous similar message [ 2470.449285] LustreError: 95775:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2470.453827] LustreError: 95775:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2470.476378] Lustre: server umount lustre-MDT0000 complete [ 2472.932473] LustreError: 93615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.955113] LustreError: 93615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2473.781947] LustreError: 95977:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2473.785223] LustreError: 95977:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2473.927845] Lustre: server umount lustre-MDT0001 complete [ 2476.926734] LustreError: 96177:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2476.930128] LustreError: 96177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2476.970296] Lustre: server umount MGS complete [ 2484.940060] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:10:29 (1763345429) [ 2508.222709] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 2516.082754] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2524.277826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.808032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2534.906829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2540.840403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2547.685215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2547.747518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.804265] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2561.804165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2561.893154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2562.166661] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2562.213600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2562.281660] Lustre: lustre-MDT0001: new disk, initializing [ 2562.375710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2562.415091] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2562.422297] Lustre: 100694:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2562.427029] LustreError: 100694:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2566.208601] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2575.147992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2575.221413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2575.380984] Lustre: lustre-OST0000: new disk, initializing [ 2575.392328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2575.447545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.552945] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2584.100461] Lustre: 101951:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ddf2c323 x1849001604324608/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763345545 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2585.087933] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2589.937492] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.949936] Lustre: MGS: Client 18a601a9-5b77-45eb-b2da-efe348087c52 (at 0@lo) reconnecting [ 2589.978597] Lustre: 99980:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2589.991355] LustreError: 99980:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2590.014599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2590.060728] Lustre: lustre-MDT0000: new disk, initializing [ 2590.197957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.222276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2591.265872] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2591.275019] Lustre: 102406:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000007bbfbaf1 x1849001604330368/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763345552 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2592.894198] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2600.368105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2600.371838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2600.461411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2620.898634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2620.905649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.914982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2620.919703] Lustre: Skipped 1 previous similar message [ 2624.994486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2625.003136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.015836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.489015] LustreError: 102928:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2626.497099] LustreError: 102928:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2626.621238] Lustre: server umount lustre-OST0000 complete [ 2635.749535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2635.752270] Lustre: Skipped 5 previous similar messages [ 2635.851860] LustreError: 103129:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2635.856557] LustreError: 103129:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2635.902153] Lustre: server umount lustre-MDT0000 complete [ 2645.861184] Lustre: server umount lustre-MDT0001 complete [ 2649.702685] Lustre: server umount MGS complete [ 2661.348926] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:13:25 (1763345605) [ 2678.809564] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2681.360773] Key type lgssc unregistered [ 2681.620581] LNet: 104871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.626373] LNetError: 104871:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2681.639123] LNet: Removed LNI 192.168.201.156@tcp [ 2682.185180] Key type .llcrypt unregistered [ 2682.187995] Key type ._llcrypt unregistered [ 2694.956798] Key type ._llcrypt registered [ 2694.958367] Key type .llcrypt registered [ 2695.065960] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2695.856771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2695.864799] alg: No test for adler32 (adler32-zlib) [ 2696.912774] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 2697.136774] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2698.824322] Key type lgssc registered [ 2699.923461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2706.098617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2706.116517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.443389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.054413] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2716.906322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2717.269324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.522352] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2725.017971] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2728.473529] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2735.525949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2735.958589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.944330] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2748.497897] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2753.396113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2762.721295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.737492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.763374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2763.745234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2763.748821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.778913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2763.800316] Lustre: Skipped 1 previous similar message [ 2767.108110] LustreError: 108323:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2767.225703] Lustre: server umount lustre-OST0000 complete [ 2773.475521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2773.483819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.496739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.106161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.109029] Lustre: Skipped 1 previous similar message [ 2784.228250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2784.240811] Lustre: Skipped 1 previous similar message [ 2785.248153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2785.350823] LustreError: 108525:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2785.357849] LustreError: 108525:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2785.415254] Lustre: server umount lustre-MDT0000 complete [ 2788.627170] LustreError: 106036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345733 with bad export cookie 12142004273698803710 [ 2788.629102] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.643928] LustreError: 106036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2788.767058] LustreError: 108729:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2788.770583] LustreError: 108729:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2788.909053] Lustre: server umount lustre-MDT0001 complete [ 2798.166805] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2800.786067] Key type lgssc unregistered [ 2801.063528] LNet: 109260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2801.078979] LNetError: 109260:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2801.088694] LNet: Removed LNI 192.168.201.156@tcp [ 2801.971967] Key type .llcrypt unregistered [ 2801.974909] Key type ._llcrypt unregistered [ 2816.021600] Key type ._llcrypt registered [ 2816.023023] Key type .llcrypt registered [ 2816.109429] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2816.941377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2817.054586] alg: No test for adler32 (adler32-zlib) [ 2818.174836] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 2818.352128] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2820.000187] Key type lgssc registered [ 2821.003750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2826.496316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2826.499769] LustreError: 110319:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2826.504958] LustreError: 110326:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2826.515519] LustreError: 110319:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 2826.526735] LustreError: 110319:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2826.543681] LustreError: 110319:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2826.546752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.099960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.693511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2835.086943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2838.644739] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2842.989882] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing lsmod [ 2845.520204] LustreError: 111499:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2845.539269] LustreError: 111502:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2845.544486] LustreError: 111499:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 2845.549884] LustreError: 111499:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2845.557689] LustreError: 111499:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2845.569864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2845.879228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.448692] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.451127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.473127] Lustre: Skipped 1 previous similar message [ 2853.860993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2853.871853] Lustre: Skipped 4 previous similar messages [ 2854.698825] LustreError: 111693:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2854.785412] Lustre: server umount lustre-MDT0000 complete [ 2857.694626] LustreError: 110341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345803 with bad export cookie 8519835001730217484 [ 2857.699970] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.704497] LustreError: 110341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2857.714162] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.714725] LustreError: 110362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.714733] LustreError: 110362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2857.737444] Lustre: Skipped 1 previous similar message [ 2857.744526] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2857.853254] LustreError: 111895:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2857.867699] LustreError: 111895:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2858.057642] Lustre: server umount lustre-MDT0001 complete [ 2861.100779] LustreError: 112098:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2861.103851] LustreError: 112098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2861.213707] Lustre: server umount lustre-OST0000 complete [ 2868.693795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.132762] LustreError: 112574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.245087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.774684] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2874.338272] LustreError: 112574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.461500] LustreError: 112574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.156917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2881.407837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2885.976298] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2891.665051] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2895.784318] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2901.667966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2901.916504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.390477] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2911.068115] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2914.230825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2921.442228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2921.445703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.473871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2924.515522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2924.538871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.656537] LustreError: 114821:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2926.679419] LustreError: 114821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2926.812864] Lustre: server umount lustre-OST0000 complete [ 2932.706268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2932.709890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.715754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2932.718198] Lustre: Skipped 2 previous similar messages [ 2936.355432] LustreError: 115023:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2936.365391] LustreError: 115023:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2936.431330] Lustre: server umount lustre-MDT0000 complete [ 2939.199732] LustreError: 112555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345884 with bad export cookie 8519835001730218177 [ 2939.209641] LustreError: 112555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2939.212793] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2939.498159] Lustre: server umount lustre-MDT0001 complete [ 2948.720855] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2951.404554] Key type lgssc unregistered [ 2951.702490] LNet: 115757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.709341] LNetError: 115757:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.743132] LNet: Removed LNI 192.168.201.156@tcp [ 2953.394453] Key type .llcrypt unregistered [ 2953.397056] Key type ._llcrypt unregistered [ 2973.066361] Key type ._llcrypt registered [ 2973.067919] Key type .llcrypt registered [ 2973.152665] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2973.919101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2973.992699] alg: No test for adler32 (adler32-zlib) [ 2975.062137] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 2975.340644] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2977.016219] Key type lgssc registered [ 2978.152717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2985.559045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2985.584369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.079991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.641303] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2997.217520] LustreError: 117148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.240704] LustreError: 117148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2998.242214] LustreError: 117148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.020277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2999.352868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3002.947402] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3008.400499] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3011.850659] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3017.958299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3018.239212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.593438] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3028.542540] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3033.592760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3039.147938] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3042.540333] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3048.029433] LustreError: 119494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3048.270822] Lustre: server umount lustre-OST0000 complete [ 3054.050229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.057383] Lustre: Skipped 1 previous similar message [ 3054.059425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3054.077175] Lustre: Skipped 1 previous similar message [ 3059.171144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3059.176797] Lustre: Skipped 1 previous similar message [ 3062.250463] LustreError: 119699:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3062.253623] LustreError: 119699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3062.329855] Lustre: server umount lustre-MDT0000 complete [ 3064.290431] LustreError: 118537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.248779] Lustre: 116461:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763345994/real 1763345994] req@000000002584553a x1849002211442560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763346010 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3065.287502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.980716] LustreError: 117130:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346011 with bad export cookie 17350625677562714029 [ 3065.987299] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.996292] LustreError: 117130:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3066.124429] LustreError: 119904:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3066.131532] LustreError: 119904:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3066.274241] Lustre: server umount lustre-MDT0001 complete [ 3076.955859] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3079.671782] Key type lgssc unregistered [ 3079.953254] LNet: 120436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3079.970295] LNetError: 120436:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3079.999252] LNet: Removed LNI 192.168.201.156@tcp [ 3080.709616] Key type .llcrypt unregistered [ 3080.711135] Key type ._llcrypt unregistered [ 3098.024049] Key type ._llcrypt registered [ 3098.035500] Key type .llcrypt registered [ 3098.157980] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3099.465867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3099.916157] alg: No test for adler32 (adler32-zlib) [ 3101.004669] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 3101.328337] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3103.040207] Key type lgssc registered [ 3104.553614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3113.057614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3113.092783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.638289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.958241] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3124.196163] LustreError: 121638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.211062] LustreError: 121638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3128.202517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3128.497161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3132.163653] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3137.009664] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3140.395767] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3146.967422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3147.226489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.417833] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3157.621649] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3159.541265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3172.950886] LustreError: 123908:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3173.074468] Lustre: server umount lustre-OST0000 complete [ 3180.007515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.011309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3180.012913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.095487] Lustre: Skipped 1 previous similar message [ 3185.127771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.130349] Lustre: Skipped 1 previous similar message [ 3187.181506] LustreError: 124109:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3187.184016] LustreError: 124109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3187.349851] Lustre: server umount lustre-MDT0000 complete [ 3190.242763] LustreError: 122846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.273669] LustreError: 122846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3190.420950] Lustre: 120947:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346119/real 1763346119] req@000000009948ae03 x1849002343557120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763346135 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3190.445976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.267253] Lustre: 120950:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346120/real 1763346120] req@00000000fb5eb684 x1849002343557760/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763346136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3192.314148] LustreError: 121617:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346137 with bad export cookie 1588512657516186947 [ 3192.317545] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.338154] LustreError: 121617:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3192.566299] LustreError: 124313:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3192.569439] LustreError: 124313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3192.759211] Lustre: server umount lustre-MDT0001 complete [ 3204.179850] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3207.032473] Key type lgssc unregistered [ 3207.302827] LNet: 124846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3207.315378] LNetError: 124846:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3208.362771] LNet: Removed LNI 192.168.201.156@tcp [ 3209.308834] Key type .llcrypt unregistered [ 3209.310821] Key type ._llcrypt unregistered [ 3236.048225] Key type ._llcrypt registered [ 3236.050385] Key type .llcrypt registered [ 3236.168948] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3237.600953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3237.669385] alg: No test for adler32 (adler32-zlib) [ 3238.853422] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 3239.194714] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3240.984189] Key type lgssc registered [ 3242.181806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3249.074263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3249.094142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.496825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.708477] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3255.777752] LustreError: 126242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.807636] LustreError: 126242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3259.878528] LustreError: 126242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.954316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3264.474477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.499346] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3274.844735] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3279.004816] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3286.332425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3286.627531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.166769] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3294.902411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3297.506813] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3303.815485] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3307.205650] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3307.210129] Lustre: Skipped 2 previous similar messages [ 3311.484976] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3311.490261] Lustre: Skipped 2 previous similar messages [ 3312.186196] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3312.223348] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3318.028588] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.56@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3318.034900] Lustre: Skipped 2 previous similar messages [ 3320.422543] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.56@tcp 192.168.0.[2,10]@tcp [ 3327.424703] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.56/24@tcp in log lustre-MDT0000 [ 3327.434267] Lustre: Skipped 2 previous similar messages [ 3334.716699] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.56/24@tcp [ 3334.720718] Lustre: Skipped 1 previous similar message [ 3341.283543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3341.285394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.298974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3341.305405] Lustre: Skipped 1 previous similar message [ 3341.321686] Lustre: Skipped 1 previous similar message [ 3342.905104] LustreError: 129749:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3343.065382] Lustre: server umount lustre-OST0000 complete [ 3351.521333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.524973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.543476] Lustre: Skipped 1 previous similar message [ 3351.553796] Lustre: Skipped 1 previous similar message [ 3352.866844] LustreError: 129950:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3352.873475] LustreError: 129950:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3352.993332] Lustre: server umount lustre-MDT0000 complete [ 3356.643942] LustreError: 126243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.124232] LustreError: 126223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346302 with bad export cookie 10643814736477636614 [ 3357.131672] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.137743] LustreError: 126223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3357.332742] LustreError: 130152:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3357.374819] LustreError: 130152:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3357.671317] Lustre: server umount lustre-MDT0001 complete [ 3372.134562] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3375.605811] Key type lgssc unregistered [ 3375.878082] LNet: 130683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3375.884642] LNetError: 130683:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3376.935360] LNet: Removed LNI 192.168.201.156@tcp [ 3377.687059] Key type .llcrypt unregistered [ 3377.688654] Key type ._llcrypt unregistered [ 3408.535978] Key type ._llcrypt registered [ 3408.537921] Key type .llcrypt registered [ 3408.695257] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3409.598979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3409.675674] alg: No test for adler32 (adler32-zlib) [ 3410.830353] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 3411.141832] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3412.816281] Key type lgssc registered [ 3414.228812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3422.455271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3422.477895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.026920] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3424.055481] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.56/24@tcp [ 3424.068172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.277909] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3434.465233] LustreError: 132226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3434.479954] LustreError: 132226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3436.158844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3436.478336] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3436.520775] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.56/24@tcp [ 3436.602829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3440.673409] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3446.236493] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3450.331702] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3457.507577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3457.960534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3463.489419] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3468.712503] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3469.262749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3481.571171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3481.577114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.590522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3482.094467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3482.107486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.116951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3484.650691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3484.653542] Lustre: Skipped 1 previous similar message [ 3487.688219] LustreError: 134529:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3487.874283] Lustre: server umount lustre-OST0000 complete [ 3492.833332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3492.836787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.852575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3494.890647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.727035] LustreError: 134733:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3497.736858] LustreError: 134733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3497.879701] Lustre: server umount lustre-MDT0000 complete [ 3500.002569] LustreError: 133851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3500.022275] LustreError: 133851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3501.312887] LustreError: 132208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346446 with bad export cookie 6499316125640734614 [ 3501.316614] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3501.325661] LustreError: 132208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3501.486837] LustreError: 134934:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3501.490898] LustreError: 134934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3501.653700] Lustre: server umount lustre-MDT0001 complete [ 3511.061101] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3513.485927] Key type lgssc unregistered [ 3513.736521] LNet: 135466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3513.743288] LNetError: 135466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3513.766290] LNet: Removed LNI 192.168.201.156@tcp [ 3514.437570] Key type .llcrypt unregistered [ 3514.439316] Key type ._llcrypt unregistered [ 3536.606045] Key type ._llcrypt registered [ 3536.607851] Key type .llcrypt registered [ 3536.708518] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3550.027550] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3550.742103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3550.801958] alg: No test for adler32 (adler32-zlib) [ 3551.842395] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 3552.042341] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3553.720176] Key type lgssc registered [ 3554.855749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3562.576684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.382747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3573.952494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3579.248458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3584.873532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3584.942440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3584.960297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.154134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3586.178958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3586.245377] Lustre: lustre-MDT0000: new disk, initializing [ 3586.330177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3586.359757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3589.690799] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3600.602540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3600.695821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3600.818289] Lustre: 138841:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3600.908833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3600.919175] Lustre: Skipped 1 previous similar message [ 3601.097668] Lustre: lustre-MDT0001: new disk, initializing [ 3601.256612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3601.300625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3601.317842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3605.373681] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3613.241185] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3616.687996] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3622.459601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3622.519355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3622.736040] Lustre: lustre-OST0000: new disk, initializing [ 3622.739253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3622.812826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.989527] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3629.123244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3629.133095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3629.262103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3635.387995] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3639.570723] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3639.770173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3644.758252] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3645.110351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3650.022941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.023105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.027352] LustreError: Skipped 1 previous similar message [ 3650.028379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.032523] Lustre: Skipped 1 previous similar message [ 3653.365812] LustreError: 141778:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3653.506561] Lustre: server umount lustre-OST0000 complete [ 3657.699279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3657.709944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.731291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3657.744102] Lustre: Skipped 1 previous similar message [ 3660.260386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.266583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.294941] Lustre: Skipped 1 previous similar message [ 3663.493396] LustreError: 141980:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3663.506416] LustreError: 141980:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3663.615807] Lustre: server umount lustre-MDT0000 complete [ 3668.131083] LustreError: 138833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346613 with bad export cookie 10664504063383906685 [ 3668.140470] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.150926] LustreError: 138833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3668.307924] LustreError: 142182:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3668.316839] LustreError: 142182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3668.521784] Lustre: server umount lustre-MDT0001 complete [ 3677.094391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.408228] LustreError: 142660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.423096] LustreError: 142660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3677.475708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.571539] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3682.789659] LustreError: 142660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.885799] LustreError: 142660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.165257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3688.492803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3691.580555] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3695.495494] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3697.883888] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3702.400104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3702.629130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.608758] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3711.156298] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3715.376570] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3722.889995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3725.631323] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3725.774590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3733.370452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3733.439810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3733.591789] Lustre: lustre-OST0001: new disk, initializing [ 3733.595912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3733.707484] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3738.631237] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3739.708049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3739.715928] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3739.819777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3745.002334] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3745.171863] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3748.559298] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3748.770676] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3751.743860] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b7b09521000.ost_server_uuid 50 [ 3753.031268] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b7b09521000.ost_server_uuid in FULL state after 0 sec [ 3760.610585] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3760.616248] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.619268] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3760.621310] Lustre: Skipped 1 previous similar message [ 3764.855713] LustreError: 146959:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3764.861665] LustreError: 146959:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3764.908239] Lustre: server umount lustre-OST0001 complete [ 3765.734450] LustreError: 144258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3765.746742] LustreError: 144258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3768.293449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3768.301498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.323476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.326777] Lustre: Skipped 1 previous similar message [ 3770.868784] LustreError: 146137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3770.876327] LustreError: 146137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3781.091945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3781.097604] Lustre: Skipped 4 previous similar messages [ 3781.098626] LustreError: 146137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.111700] LustreError: 146137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3782.112280] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3782.151296] LustreError: 147160:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3782.155770] LustreError: 147160:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3782.269077] Lustre: server umount lustre-OST0000 complete [ 3785.700809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3785.706140] LustreError: Skipped 1 previous similar message [ 3785.708662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.714630] Lustre: Skipped 1 previous similar message [ 3791.226368] LustreError: 147364:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3791.259585] LustreError: 147364:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3791.315489] Lustre: server umount lustre-MDT0000 complete [ 3794.995135] LustreError: 142639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346740 with bad export cookie 10664504063383907567 [ 3794.998702] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3795.018680] LustreError: 142639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3795.369927] Lustre: server umount lustre-MDT0001 complete [ 3812.195578] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3814.683892] Key type lgssc unregistered [ 3814.978987] LNet: 148557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3815.002949] LNetError: 148557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3815.024073] LNet: Removed LNI 192.168.201.156@tcp [ 3815.926495] Key type .llcrypt unregistered [ 3815.928578] Key type ._llcrypt unregistered [ 3843.120231] Key type ._llcrypt registered [ 3843.121870] Key type .llcrypt registered [ 3843.232424] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3854.914933] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3855.720688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3855.730316] alg: No test for adler32 (adler32-zlib) [ 3856.742250] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 3856.925800] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3858.594061] Key type lgssc registered [ 3859.675098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3868.312171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3874.489782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3880.576804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3886.278604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3891.938152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3892.005654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3892.021662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.294588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3893.319694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3893.376931] Lustre: lustre-MDT0000: new disk, initializing [ 3893.459889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3893.486731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3896.460424] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3905.587522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3905.648518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3905.728704] Lustre: 152485:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3905.789789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3905.793175] Lustre: Skipped 1 previous similar message [ 3905.851374] Lustre: lustre-MDT0001: new disk, initializing [ 3905.947329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3905.996384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3906.006558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3909.687430] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3916.464241] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3919.412416] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3925.122996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3925.199531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3925.434296] Lustre: lustre-OST0000: new disk, initializing [ 3925.438878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3925.511347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.193942] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3935.811685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3935.824672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3935.943488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3938.139742] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3941.342112] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3941.523598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3945.139416] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3945.321603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3951.586327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.596158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3951.597076] Lustre: Skipped 1 previous similar message [ 3956.709076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.716554] Lustre: Skipped 2 previous similar messages [ 3961.312556] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3961.339127] LustreError: 155422:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3961.413881] Lustre: server umount lustre-OST0000 complete [ 3966.947238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.961873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3966.968149] Lustre: Skipped 1 previous similar message [ 3970.843567] LustreError: 155625:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3970.851213] LustreError: 155625:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3970.963409] Lustre: server umount lustre-MDT0000 complete [ 3974.500134] LustreError: 152476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346919 with bad export cookie 25344746586336348 [ 3974.523779] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3974.678367] LustreError: 155826:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3974.706361] LustreError: 155826:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3975.004175] Lustre: server umount lustre-MDT0001 complete [ 3983.685682] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 21:35:28 (1763346928) [ 3997.422453] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 4006.033210] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4013.682832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4018.860383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4024.033318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4029.927097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4035.538813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4035.598485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.751214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4035.785726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4035.845748] Lustre: lustre-MDT0000: new disk, initializing [ 4035.924578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.949820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4038.907883] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4047.401829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4047.475577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4047.559480] Lustre: 158819:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4047.602364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4047.604860] Lustre: Skipped 1 previous similar message [ 4047.665493] Lustre: lustre-MDT0001: new disk, initializing [ 4047.751196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4047.791450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4047.797766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4050.612897] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4056.778319] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4059.923899] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4065.182287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4065.247792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4065.420870] Lustre: lustre-OST0000: new disk, initializing [ 4065.424315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4065.497241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.488500] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4072.531695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4072.546710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4072.645587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4077.229676] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4080.975568] Lustre: Failing over lustre-OST0000 [ 4081.025096] LustreError: 161257:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4081.032240] LustreError: 161257:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4081.148919] Lustre: server umount lustre-OST0000 complete [ 4099.555301] Lustre: 150700:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347028/real 1763347028] req@00000000a0569969 x1849003136323328/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763347044 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4099.598811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.600759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4099.624677] Lustre: Skipped 1 previous similar message [ 4099.955175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.970557] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4100.455515] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4100.953971] Lustre: 150697:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347028/real 1763347028] req@00000000af2de1d8 x1849003136324096/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1763347044 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4104.100147] Lustre: 150697:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347033/real 1763347033] req@00000000c64ba686 x1849003136325120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763347049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4104.125664] Lustre: 150697:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4105.407436] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4109.157880] Lustre: 150699:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347038/real 1763347038] req@000000005824e4b2 x1849003136326400/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763347054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4109.197061] Lustre: lustre-OST0000: Recovery over after 0:09, of 3 clients 3 recovered and 0 were evicted. [ 4109.206974] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4109.666882] Lustre: Failing over lustre-MDT0000 [ 4109.791095] LustreError: 162238:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4109.801286] LustreError: 162238:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4109.873277] Lustre: server umount lustre-MDT0000 complete [ 4110.688475] LustreError: 158825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.717167] LustreError: 158825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4114.336161] Lustre: 150700:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347043/real 1763347043] req@000000009b5ce6f4 x1849003136327424/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763347059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4114.368125] Lustre: 150700:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4114.403629] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.404190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4114.431797] LustreError: 158831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.433629] Lustre: Skipped 2 previous similar messages [ 4115.766365] LustreError: 158826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.784262] LustreError: 158826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4119.522474] LustreError: 158827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4119.534064] LustreError: 158827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4124.644588] LustreError: 159720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4124.669837] LustreError: 159720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4128.372237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4128.471717] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4128.776940] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4132.149252] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4132.759851] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4133.906800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4133.913441] Lustre: Skipped 1 previous similar message [ 4133.920851] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4133.974990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4144.103189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.108547] Lustre: Skipped 1 previous similar message [ 4144.111280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4144.133708] Lustre: Skipped 1 previous similar message [ 4146.753126] LustreError: 163125:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4146.765332] LustreError: 163125:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4146.970597] Lustre: server umount lustre-OST0000 complete [ 4154.343570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.349039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4154.376345] Lustre: Skipped 1 previous similar message [ 4154.388897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4154.396290] Lustre: Skipped 1 previous similar message [ 4158.247358] Lustre: server umount lustre-MDT0000 complete [ 4159.457205] LustreError: 158831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.498636] LustreError: 158831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4162.029391] LustreError: 158812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347107 with bad export cookie 25344746586338462 [ 4162.034153] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4162.043341] LustreError: 158812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4162.358852] Lustre: server umount lustre-MDT0001 complete [ 4172.516874] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4174.969549] Key type lgssc unregistered [ 4175.251710] LNet: 164059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4175.270553] LNetError: 164059:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4175.288504] LNet: Removed LNI 192.168.201.156@tcp [ 4176.044782] Key type .llcrypt unregistered [ 4176.047491] Key type ._llcrypt unregistered [ 4209.744184] Key type ._llcrypt registered [ 4209.745850] Key type .llcrypt registered [ 4209.861646] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4211.480936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4211.501184] alg: No test for adler32 (adler32-zlib) [ 4212.604149] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 4212.904799] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4214.632220] Key type lgssc registered [ 4215.605097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4224.361302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4224.379840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.911612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.845902] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4236.259264] LustreError: 165561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.280531] LustreError: 165561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4237.448110] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4241.404568] LustreError: 165561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.514981] LustreError: 165590:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4241.520731] LustreError: 165590:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4241.536071] LustreError: 165590:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4241.666699] LustreError: 166115:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4241.943830] Lustre: server umount lustre-MDT0000 complete [ 4257.272520] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 4267.094694] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4277.308934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4285.556202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4293.419342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4300.386270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4307.554894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4307.706912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.940194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4307.973890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4308.024200] Lustre: lustre-MDT0000: new disk, initializing [ 4308.127185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4308.148732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4311.936473] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4322.615103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4322.681845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.779261] Lustre: 168760:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4322.847883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4322.850951] Lustre: Skipped 1 previous similar message [ 4322.923904] Lustre: lustre-MDT0001: new disk, initializing [ 4323.043338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.089476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4323.101460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4326.447069] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4333.241206] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4336.666775] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4342.744157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.819566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4342.983978] Lustre: lustre-OST0000: new disk, initializing [ 4342.989613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4343.074825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.742412] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4349.029964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4349.034539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4349.151075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4355.796264] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4360.059948] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4362.009550] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:41:46 (1763347306) [ 4610.019853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4610.042179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.070118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4611.721652] LustreError: 172290:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4611.728522] LustreError: 172290:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4611.902769] Lustre: server umount lustre-MDT0000 complete [ 4615.650104] LustreError: 168773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.667054] LustreError: 168773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4615.925495] LustreError: 168754:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347561 with bad export cookie 5415875123026182013 [ 4615.945577] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4620.769256] LustreError: 172162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.774482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.852374] Lustre: Skipped 2 previous similar messages [ 4620.882132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4620.889804] Lustre: Skipped 2 previous similar messages [ 4622.118389] LustreError: 172493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4622.126435] LustreError: 172493:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4622.297444] Lustre: server umount lustre-MDT0001 complete [ 4633.215090] LustreError: 172695:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4633.225908] LustreError: 172695:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4633.458156] Lustre: server umount lustre-OST0000 complete [ 4643.245299] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 4654.942860] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4668.770273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4676.786754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4684.774549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4692.436654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4700.550289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4700.651849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4700.854742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4700.885900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4700.967638] Lustre: lustre-MDT0000: new disk, initializing [ 4701.053551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4701.089154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4705.975907] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4716.620549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4716.700579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4716.799400] Lustre: 175049:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4716.860469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4716.869394] Lustre: Skipped 1 previous similar message [ 4716.932320] Lustre: lustre-MDT0001: new disk, initializing [ 4717.050514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4717.095509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4717.110141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4721.037782] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4728.456691] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4732.382771] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4739.386546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4739.471615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4739.658713] Lustre: lustre-OST0000: new disk, initializing [ 4739.667826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4739.753802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.263665] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4746.840965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4746.855879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4747.008039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4753.299149] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4757.851261] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4758.091544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4762.774106] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4763.017239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4767.718230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.719945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4767.752444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4767.755608] Lustre: Skipped 1 previous similar message [ 4771.267741] LustreError: 177987:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4771.270828] LustreError: 177987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4771.541819] Lustre: server umount lustre-OST0000 complete [ 4777.953668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.967171] Lustre: Skipped 1 previous similar message [ 4777.972059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.981290] Lustre: Skipped 1 previous similar message [ 4781.608261] LustreError: 178189:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4781.612641] LustreError: 178189:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4781.829595] Lustre: server umount lustre-MDT0000 complete [ 4783.078863] LustreError: 175056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.123298] LustreError: 175056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4786.257102] LustreError: 175040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347731 with bad export cookie 5415875123026340836 [ 4786.262710] LustreError: 175040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4786.269271] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4786.723403] Lustre: server umount lustre-MDT0001 complete [ 4795.303780] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:48:59 (1763347739) [ 4811.048950] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 4820.844291] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4831.471818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4838.276427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4844.989173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4852.230952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4860.862786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4860.996086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.141484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4861.165510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4861.239940] Lustre: lustre-MDT0000: new disk, initializing [ 4861.396915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.413806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4866.272673] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4879.169642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4879.256097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4879.392385] Lustre: 181386:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4879.723529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4884.315856] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4893.195978] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4897.873704] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4906.130301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4906.216544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4906.528539] Lustre: lustre-OST0000: new disk, initializing [ 4906.530652] Lustre: Skipped 1 previous similar message [ 4906.532909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4906.535763] Lustre: Skipped 2 previous similar messages [ 4912.642965] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4914.226129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4914.233658] Lustre: Skipped 1 previous similar message [ 4914.243181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4914.395207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4920.134545] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4930.022249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4930.026957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.041430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4930.094931] Lustre: Skipped 2 previous similar messages [ 4933.329539] LustreError: 183932:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4933.334391] LustreError: 183932:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4933.474602] Lustre: server umount lustre-OST0000 complete [ 4940.257741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.284913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4940.287676] Lustre: Skipped 1 previous similar message [ 4943.767641] Lustre: server umount lustre-MDT0000 complete [ 4945.379448] LustreError: 181391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.407968] LustreError: 181391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4947.390794] LustreError: 181378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347892 with bad export cookie 5415875123026341718 [ 4947.401321] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.426347] LustreError: 181378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4947.686330] Lustre: server umount lustre-MDT0001 complete [ 4956.995049] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:51:41 (1763347901) [ 4972.755123] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 4980.946527] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4991.119432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4997.522215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5004.738723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5011.907047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5019.646394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5019.709266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.907720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5019.931418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5020.012538] Lustre: lustre-MDT0000: new disk, initializing [ 5020.153884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5020.157188] Lustre: Skipped 2 previous similar messages [ 5020.215823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5024.546818] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5036.656690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5036.722378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5036.840967] Lustre: 187330:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5037.132160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5041.144219] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5048.994478] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5053.141079] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5059.940481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5060.036422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5065.903539] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5069.876535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5070.044062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5072.661997] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5080.548099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5080.561394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5080.570672] Lustre: Skipped 1 previous similar message [ 5080.574187] LustreError: Skipped 1 previous similar message [ 5080.575043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.276115] LustreError: 189879:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5085.278844] LustreError: 189879:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5085.489048] Lustre: server umount lustre-OST0000 complete [ 5095.566960] Lustre: server umount lustre-MDT0000 complete [ 5095.906550] LustreError: 189229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5095.920337] LustreError: 189229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5099.439351] LustreError: 187320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348044 with bad export cookie 5415875123026342810 [ 5099.443049] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5099.460560] LustreError: 187320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5099.818659] Lustre: server umount lustre-MDT0001 complete [ 5111.846792] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5115.144883] Key type lgssc unregistered [ 5115.443910] LNet: 190815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5115.471783] LNetError: 190815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5116.522789] LNet: Removed LNI 192.168.201.156@tcp [ 5117.507900] Key type .llcrypt unregistered [ 5117.509532] Key type ._llcrypt unregistered [ 5141.997332] Key type ._llcrypt registered [ 5141.999617] Key type .llcrypt registered [ 5142.102868] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5143.391080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5143.478976] alg: No test for adler32 (adler32-zlib) [ 5144.668242] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 5144.954780] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5146.624247] Key type lgssc registered [ 5147.763392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5155.848350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5155.861949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.435285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.623812] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5167.590183] LustreError: 192212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.604672] LustreError: 192212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5168.985397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5169.316867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5173.143305] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5178.465486] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5181.890929] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5188.696681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5189.077879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.797257] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5199.916425] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5202.419151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5222.882307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5222.885809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.912187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5222.920605] Lustre: Skipped 1 previous similar message [ 5226.895291] LustreError: 194482:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5227.117268] Lustre: server umount lustre-OST0000 complete [ 5233.121599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.130526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.148577] Lustre: Skipped 2 previous similar messages [ 5238.244890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5238.247213] Lustre: Skipped 2 previous similar messages [ 5243.370276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5243.382928] Lustre: Skipped 1 previous similar message [ 5245.409220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5245.492219] LustreError: 194684:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5245.496443] LustreError: 194684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5245.600782] Lustre: server umount lustre-MDT0000 complete [ 5248.489176] LustreError: 192208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5249.180498] LustreError: 192194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348194 with bad export cookie 12074887947897354486 [ 5249.197962] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5249.317098] LustreError: 194888:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5249.325622] LustreError: 194888:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5249.523738] Lustre: server umount lustre-MDT0001 complete [ 5258.928740] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5261.436652] Key type lgssc unregistered [ 5261.673687] LNet: 195419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5261.680545] LNetError: 195419:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5261.695914] LNet: Removed LNI 192.168.201.156@tcp [ 5262.329828] Key type .llcrypt unregistered [ 5262.333388] Key type ._llcrypt unregistered [ 5283.970258] Key type ._llcrypt registered [ 5283.972634] Key type .llcrypt registered [ 5284.041622] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5285.304129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5285.351140] alg: No test for adler32 (adler32-zlib) [ 5286.411742] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 5286.594723] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5288.272247] Key type lgssc registered [ 5289.447695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5298.051869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5298.096647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5299.632193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.799529] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5308.906610] LustreError: 196811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5308.940079] LustreError: 196811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5312.324789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5312.607708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5316.102968] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5320.018277] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5322.497994] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5328.511722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5328.817806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5335.179775] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5340.805682] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5344.256698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5351.756308] Lustre: lustre-OST0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 5352.417264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5352.424732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.445996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5354.465469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5354.471083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5354.480695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.679284] LustreError: 199060:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5356.816662] Lustre: server umount lustre-OST0000 complete [ 5361.804614] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b7b09150800.ost_server_uuid 50 [ 5363.704405] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5378.978299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.981941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.995142] Lustre: Skipped 1 previous similar message [ 5379.032353] Lustre: Skipped 2 previous similar messages [ 5383.139760] LustreError: 199428:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5383.147802] LustreError: 199428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5383.242022] Lustre: server umount lustre-MDT0000 complete [ 5384.160950] LustreError: 197524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5384.171016] LustreError: 197524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5386.511721] LustreError: 196792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348331 with bad export cookie 8495582497522863041 [ 5386.516521] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5386.524769] LustreError: 196792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5386.661509] LustreError: 199629:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5386.672383] LustreError: 199629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5386.867726] Lustre: server umount lustre-MDT0001 complete [ 5396.351883] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:58:59 (1763348339) [ 5404.151711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.507937] LustreError: 200300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.570366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5409.271573] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5409.774766] LustreError: 200300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.928158] LustreError: 200300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.699930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5418.190468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5422.220622] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5427.713559] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5431.323293] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5438.450108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5438.908957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5445.712387] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5449.218043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5452.739955] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5459.570419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5459.662774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5459.873559] Lustre: lustre-OST0001: new disk, initializing [ 5459.885150] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5459.996489] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5465.392829] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5467.249319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5467.264703] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5467.362824] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5472.814554] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5477.867866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.869472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5477.879346] Lustre: Skipped 1 previous similar message [ 5481.544244] LustreError: 203506:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5481.559524] LustreError: 203506:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5481.645862] Lustre: server umount lustre-OST0000 complete [ 5482.989968] LustreError: 203260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.022608] LustreError: 203260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5487.613382] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5487.781623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5491.962742] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5492.136047] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5499.194751] LustreError: 202373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.211406] LustreError: 202373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5508.577586] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5508.581458] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.584550] LustreError: Skipped 1 previous similar message [ 5508.590937] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5508.592885] Lustre: Skipped 1 previous similar message [ 5508.608594] Lustre: Skipped 3 previous similar messages [ 5510.782647] LustreError: 204231:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5510.785669] LustreError: 204231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5510.894987] Lustre: server umount lustre-OST0001 complete [ 5515.745861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5518.819178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.838149] Lustre: Skipped 1 previous similar message [ 5520.500951] LustreError: 204433:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5520.504925] LustreError: 204433:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5520.597187] Lustre: server umount lustre-MDT0000 complete [ 5524.125228] LustreError: 200280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348469 with bad export cookie 8495582497522864112 [ 5524.128479] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5524.134842] LustreError: 200280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5524.550918] Lustre: server umount lustre-MDT0001 complete [ 5530.234651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5534.263163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5538.337892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5542.240708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5549.592111] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:01:33 (1763348493) [ 5555.345737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.551246] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5555.595216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5555.750661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.042799] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5566.862579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5566.951580] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5566.986739] Lustre: 206141:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5571.746420] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5577.394828] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5580.925143] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5588.044452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5588.312252] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5593.770241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5594.188562] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5599.569871] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5605.982859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5606.021717] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5612.963473] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5615.440875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5618.879920] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5624.293415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5624.303896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.325905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5624.328512] Lustre: Skipped 3 previous similar messages [ 5627.735122] LustreError: 209341:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5627.753354] LustreError: 209341:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5627.837273] Lustre: server umount lustre-OST0000 complete [ 5629.768877] LustreError: 207770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.795090] LustreError: 207770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5646.314419] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5646.318234] LustreError: Skipped 1 previous similar message [ 5648.798765] Lustre: server umount lustre-OST0001 complete [ 5656.546782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.548821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.551123] Lustre: Skipped 7 previous similar messages [ 5656.567525] Lustre: Skipped 4 previous similar messages [ 5658.445607] Lustre: server umount lustre-MDT0000 complete [ 5661.599317] LustreError: 206134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348606 with bad export cookie 8495582497522865519 [ 5661.601535] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5661.610181] LustreError: 206134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5661.759469] LustreError: 209968:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5661.764552] LustreError: 209968:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5661.936417] Lustre: server umount lustre-MDT0001 complete [ 5666.566632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5669.501590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5672.704249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5675.668588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5681.155181] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:03:45 (1763348625) [ 5695.594410] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 5706.349367] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5716.629260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5722.234475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5727.158431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5732.634621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5737.959953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5738.007496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.178187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5738.201104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5738.278933] Lustre: lustre-MDT0000: new disk, initializing [ 5738.394527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.398982] Lustre: Skipped 3 previous similar messages [ 5738.426145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5741.662804] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5750.667251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5750.739688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5750.845341] Lustre: 213793:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5750.929364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5750.932610] Lustre: Skipped 1 previous similar message [ 5751.021601] Lustre: lustre-MDT0001: new disk, initializing [ 5751.194277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5751.202715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5754.899339] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5762.118336] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5765.540869] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5770.912736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5770.976492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5771.224568] Lustre: lustre-OST0000: new disk, initializing [ 5771.227580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5775.855055] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5779.509555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5779.519881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5779.583407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5782.019157] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5785.142293] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5785.328958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5788.312102] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5788.524436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5790.183246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5790.186772] LustreError: Skipped 2 previous similar messages [ 5790.188633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.199371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5790.202670] Lustre: Skipped 2 previous similar messages [ 5796.022506] LustreError: 216732:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5796.027406] LustreError: 216732:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5796.131367] Lustre: server umount lustre-OST0000 complete [ 5805.449902] Lustre: server umount lustre-MDT0000 complete [ 5808.323317] LustreError: 213785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348753 with bad export cookie 8495582497522867304 [ 5808.332048] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5808.642616] Lustre: server umount lustre-MDT0001 complete [ 5816.058712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.410410] LustreError: 217614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.430578] LustreError: 217614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5820.319278] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5828.484488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5832.565393] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5836.590745] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5839.700418] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5845.013992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5849.619329] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5854.721820] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5857.767748] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5862.184817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 5866.197159] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5866.415690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5870.053904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5870.064240] LustreError: Skipped 1 previous similar message [ 5874.368767] Lustre: server umount lustre-OST0000 complete [ 5879.626391] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5883.224643] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5910.439425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5910.685812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.695719] Lustre: Skipped 5 previous similar messages [ 5911.721866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.743577] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5915.562193] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5920.279780] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5922.026991] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.037285] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5930.508975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec [ 5935.613723] LustreError: 222027:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5935.616836] LustreError: 222027:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5935.858504] Lustre: server umount lustre-OST0000 complete [ 5941.733495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5941.744576] LustreError: Skipped 1 previous similar message [ 5941.748609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.757071] Lustre: Skipped 5 previous similar messages [ 5941.766812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5941.770908] Lustre: Skipped 9 previous similar messages [ 5948.976893] Lustre: server umount lustre-MDT0000 complete [ 5952.215797] LustreError: 217593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348897 with bad export cookie 8495582497522868186 [ 5952.224339] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5952.228793] LustreError: 217593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5961.825412] Lustre: server umount lustre-MDT0001 complete [ 5969.518949] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:08:33 (1763348913) [ 5975.873296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5976.226470] LustreError: 223105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.242175] LustreError: 223105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5980.010686] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5989.129147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5993.082976] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5998.405540] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6001.971926] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6009.359351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6014.551420] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6018.945766] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6021.750333] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6021.906351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6025.410748] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6025.566151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6031.043854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6031.107390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6031.226444] Lustre: lustre-OST0001: new disk, initializing [ 6031.234433] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6036.518418] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6038.601887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6038.608913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6038.690890] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6043.695266] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6048.228654] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6048.410283] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6052.552075] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6052.762553] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6069.216277] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6069.331166] Lustre: server umount lustre-OST0001 complete [ 6073.456954] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6073.638548] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6076.432892] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6076.587788] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6104.727765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6109.871786] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6110.182105] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6110.192934] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6110.194565] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6115.034476] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6116.665365] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6120.425276] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6120.429051] LustreError: Skipped 2 previous similar messages [ 6124.345794] Lustre: server umount lustre-OST0001 complete [ 6153.184151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6153.296615] Lustre: server umount lustre-OST0000 complete [ 6162.449287] Lustre: server umount lustre-MDT0000 complete [ 6165.789258] LustreError: 223085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349111 with bad export cookie 8495582497522869152 [ 6165.795199] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6165.807526] LustreError: 223085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6180.327037] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6180.556804] Lustre: server umount lustre-MDT0001 complete [ 6185.405902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6188.688469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6192.208944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6196.476602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6203.560301] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:12:27 (1763349147) [ 6210.434042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.844728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6210.902823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6211.142879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.145651] Lustre: Skipped 5 previous similar messages [ 6215.019188] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6221.511237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6221.613461] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6221.634190] Lustre: 231089:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6224.908597] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6229.139417] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6232.459216] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6237.972861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6238.107178] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6242.735918] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6247.736309] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6258.928512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6258.976630] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6264.538724] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6270.764255] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6272.754501] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6276.503531] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6276.775776] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6280.884426] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6281.078326] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6284.997951] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b7b18b9a800.ost_server_uuid 50 [ 6286.599742] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b7b18b9a800.ost_server_uuid in FULL state after 0 sec [ 6287.693782] Lustre: Permanently deactivating lustre-OST0001 [ 6287.718928] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6290.829125] Lustre: Permanently reactivating lustre-OST0001 [ 6298.599509] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.622388] Lustre: Skipped 9 previous similar messages [ 6298.633891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6298.636526] Lustre: Skipped 23 previous similar messages [ 6300.578197] LustreError: 235079:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6300.601719] LustreError: 235079:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6300.710905] Lustre: server umount lustre-OST0001 complete [ 6304.226050] LustreError: 232749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.245827] LustreError: 232749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 6310.506406] Lustre: server umount lustre-OST0000 complete [ 6328.288222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6328.513232] Lustre: server umount lustre-MDT0000 complete [ 6331.932649] LustreError: 231081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349277 with bad export cookie 8495582497522870440 [ 6331.938992] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6331.950078] LustreError: 231081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6332.334557] Lustre: server umount lustre-MDT0001 complete [ 6337.275207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6340.429390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6343.595140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6346.639762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6352.512176] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:14:56 (1763349296) [ 6353.417615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.100403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.351370] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6359.373091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6359.379027] Lustre: Skipped 5 previous similar messages [ 6362.647362] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6370.295888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6370.397962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6370.433964] Lustre: 237243:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6373.899698] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6379.070413] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6382.166974] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6387.364470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.426645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6387.569400] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6387.599096] Lustre: Permanently deactivating lustre-OST0000 [ 6387.613095] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6392.099348] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6396.483739] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6396.947878] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6402.162282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6402.215950] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6405.457960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6406.929798] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6411.389829] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6412.875807] Lustre: Permanently reactivating lustre-OST0000 [ 6412.892852] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6412.907054] Lustre: Skipped 2 previous similar messages [ 6420.551431] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6420.561759] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.581922] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6420.584174] Lustre: Skipped 1 previous similar message [ 6420.627398] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6420.658140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6426.089761] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.095674] LustreError: Skipped 8 previous similar messages [ 6427.297446] Lustre: server umount lustre-OST0001 complete [ 6432.553828] Lustre: server umount lustre-OST0000 complete [ 6438.421223] Lustre: server umount lustre-MDT0000 complete [ 6441.967803] LustreError: 237234:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349387 with bad export cookie 8495582497522872092 [ 6441.980564] LustreError: 237234:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6441.981647] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.347975] Lustre: server umount lustre-MDT0001 complete [ 6451.793086] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 6453.680455] Key type lgssc unregistered [ 6453.911241] LNet: 241658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6453.919624] LNetError: 241658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6453.942956] LNet: Removed LNI 192.168.201.156@tcp [ 6454.455492] Key type .llcrypt unregistered [ 6454.457386] Key type ._llcrypt unregistered [ 6471.563072] Key type ._llcrypt registered [ 6471.571972] Key type .llcrypt registered [ 6471.674793] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6472.386850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6472.498143] alg: No test for adler32 (adler32-zlib) [ 6473.536359] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 6473.764864] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 6475.410383] Key type lgssc registered [ 6476.547798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6480.110153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6487.701994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6487.744274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.455326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6493.158185] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6498.791068] LustreError: 243091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.823267] LustreError: 243091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6501.396922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6501.479575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6501.621902] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6501.675710] Lustre: 243079:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6501.702211] Lustre: Permanently deactivating lustre-MDT0001 [ 6501.732295] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6502.153141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6507.054515] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6507.069283] LustreError: 243122:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6507.079715] LustreError: 243122:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6507.084578] LustreError: 243122:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 6507.117731] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6512.841930] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6516.065586] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6522.246235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6522.604600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6526.700977] LustreError: 244915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6526.713484] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6528.546633] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6532.066299] LustreError: 244708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.122126] LustreError: 244708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.138045] LustreError: 244708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6534.142114] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6534.142347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6539.253463] LustreError: 244708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6539.268246] LustreError: 244708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6540.734766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6540.924716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6545.118342] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6546.421530] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6546.440584] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6550.223376] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6554.224972] Lustre: Permanently reactivating lustre-MDT0001 [ 6554.262121] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6554.271141] Lustre: Skipped 2 previous similar messages [ 6557.975380] Lustre: lustre-MDT0001: Client d3d44106-0e6a-43d5-a014-f25088da916e (at 192.168.201.56@tcp) reconnecting [ 6563.026990] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6564.823952] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6569.584394] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6569.763128] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6573.921946] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6574.143630] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6576.419883] Lustre: Permanently deactivating lustre-MDT0001 [ 6576.448478] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6576.462118] Lustre: Skipped 2 previous similar messages [ 6586.504382] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6599.658152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.658735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6599.670781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.689203] Lustre: Skipped 2 previous similar messages [ 6600.980179] LustreError: 247213:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6601.142613] Lustre: server umount lustre-MDT0000 complete [ 6604.769271] LustreError: 243091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.795735] LustreError: 243091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6605.111552] LustreError: 247009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349550 with bad export cookie 6947985843214245548 [ 6605.115662] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6605.119179] LustreError: 247009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6605.233306] LustreError: 247414:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6605.236222] LustreError: 247414:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6605.435073] Lustre: server umount lustre-MDT0001 complete [ 6611.269246] LustreError: 247615:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6611.284589] LustreError: 247615:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6611.337864] Lustre: server umount lustre-OST0000 complete [ 6621.475996] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:19:25 (1763349565) [ 6630.901228] LustreError: 248620:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6630.930123] LustreError: 248620:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6631.126213] Lustre: server umount lustre-OST0001 complete [ 6636.483398] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 6644.886476] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6653.605795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6658.735552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6664.170995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6669.598262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6675.491613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6675.543859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.740590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6675.746670] Lustre: Skipped 2 previous similar messages [ 6675.787758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6675.853850] Lustre: lustre-MDT0000: new disk, initializing [ 6675.948188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6675.971505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6680.020653] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6689.031602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6689.107904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6689.171857] Lustre: 250807:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6689.238622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6689.244042] Lustre: Skipped 1 previous similar message [ 6689.361101] Lustre: lustre-MDT0001: new disk, initializing [ 6689.433965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6689.484221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6689.489499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6693.087428] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6699.332343] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6702.387601] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6708.862790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6708.934144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6709.208979] Lustre: lustre-OST0000: new disk, initializing [ 6709.212942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6709.297387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6713.579550] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6715.960830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6715.975315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6716.040177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6719.905535] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6721.736696] LustreError: 250813:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6729.069928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6729.151516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6729.277768] Lustre: lustre-OST0001: new disk, initializing [ 6729.286732] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6731.824257] LustreError: 250813:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6731.838905] LustreError: 250813:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6733.817561] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6738.005425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6738.011934] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6738.128402] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6740.217875] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6741.681472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6741.936192] LustreError: 250813:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6745.846066] LustreError: 254301:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6745.853861] LustreError: 254301:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6745.901464] Lustre: server umount lustre-OST0001 complete [ 6748.648909] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6748.649284] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.654492] LustreError: Skipped 1 previous similar message [ 6748.659977] Lustre: Skipped 1 previous similar message [ 6748.670336] LustreError: 252539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6748.684302] LustreError: 252539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6762.465453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6762.480367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6762.487780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6762.488581] Lustre: Skipped 2 previous similar messages [ 6762.499966] Lustre: Skipped 4 previous similar messages [ 6764.004878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.616169] LustreError: 254526:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6766.625838] LustreError: 254526:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6766.745731] Lustre: server umount lustre-OST0000 complete [ 6771.684051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6771.691036] LustreError: Skipped 1 previous similar message [ 6771.694357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.712858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6771.719282] Lustre: Skipped 1 previous similar message [ 6776.144484] Lustre: server umount lustre-MDT0000 complete [ 6779.199232] LustreError: 250798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349724 with bad export cookie 6947985843214248369 [ 6779.204916] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6779.219532] LustreError: 250798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6785.459371] LustreError: 254929:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6785.463431] LustreError: 254929:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6785.647702] Lustre: server umount lustre-MDT0001 complete [ 6792.764411] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 6795.070549] Key type lgssc unregistered [ 6795.329841] LNet: 255461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6795.340461] LNetError: 255461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6795.356309] LNet: Removed LNI 192.168.201.156@tcp [ 6795.995688] Key type .llcrypt unregistered [ 6795.999040] Key type ._llcrypt unregistered [ 6821.093989] Key type ._llcrypt registered [ 6821.095429] Key type .llcrypt registered [ 6821.186694] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 6833.410342] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6834.390549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6834.512897] alg: No test for adler32 (adler32-zlib) [ 6835.581991] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 6835.754773] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 6837.384203] Key type lgssc registered [ 6838.314046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6845.858490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6850.690570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6855.812891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6861.619718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6867.794222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6867.857956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6867.886819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.056921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6869.079675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6869.140552] Lustre: lustre-MDT0000: new disk, initializing [ 6869.229346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.247441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6872.363102] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6881.143325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6881.186239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6881.259698] Lustre: 259391:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6881.303883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6881.307060] Lustre: Skipped 1 previous similar message [ 6881.363569] Lustre: lustre-MDT0001: new disk, initializing [ 6881.421709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6881.446906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6881.452957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6884.168709] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6890.152959] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6893.308684] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6900.536987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6900.652953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6901.115218] Lustre: lustre-OST0000: new disk, initializing [ 6901.118470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6901.181960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.815226] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6910.089723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6910.107891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6910.408174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6917.258816] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6922.027588] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6922.258735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6925.936497] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6926.155194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6930.913373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6930.915619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.924407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6930.936189] Lustre: Skipped 1 previous similar message [ 6930.946629] Lustre: Skipped 1 previous similar message [ 6936.038807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6936.042102] Lustre: Skipped 1 previous similar message [ 6941.156420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6941.173470] Lustre: Skipped 1 previous similar message [ 6942.176117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6942.226352] LustreError: 262327:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6942.376885] Lustre: server umount lustre-OST0000 complete [ 6946.276017] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6946.286394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.295517] Lustre: Skipped 1 previous similar message [ 6946.328236] Lustre: Skipped 1 previous similar message [ 6951.399317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6951.402188] Lustre: Skipped 1 previous similar message [ 6951.687830] LustreError: 262529:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6951.691118] LustreError: 262529:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6951.820544] Lustre: server umount lustre-MDT0000 complete [ 6955.108650] LustreError: 262530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349900 with bad export cookie 5257721290990416021 [ 6955.113703] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6955.125267] LustreError: 262530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6955.274209] LustreError: 262732:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6955.282476] LustreError: 262732:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6955.455093] Lustre: server umount lustre-MDT0001 complete [ 6963.138992] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 22:25:07 (1763349907) [ 6969.163942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.594713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6973.263910] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6974.947181] LustreError: 263404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.960642] LustreError: 263404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6979.046901] LustreError: 263404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6980.077175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6980.369197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6983.592400] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6988.133920] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6991.941344] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6998.364191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6998.685730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7003.336075] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7008.340273] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7060.450598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7060.455042] LustreError: Skipped 1 previous similar message [ 7060.457757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.479490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7060.485791] Lustre: Skipped 1 previous similar message [ 7065.153257] LustreError: 265733:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7065.161674] LustreError: 265733:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7065.316527] Lustre: server umount lustre-OST0000 complete [ 7069.242751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7081.859177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7082.234361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7087.050993] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7092.098222] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7100.584210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7100.589970] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7100.590309] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7100.599900] Lustre: Skipped 1 previous similar message [ 7136.740646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7136.742860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.744583] LustreError: Skipped 1 previous similar message [ 7136.744809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7136.753119] Lustre: Skipped 2 previous similar messages [ 7137.914090] LustreError: 267406:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7137.917043] LustreError: 267406:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7138.021984] Lustre: server umount lustre-OST0000 complete [ 7141.858281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.868382] Lustre: Skipped 1 previous similar message [ 7147.181126] LustreError: 267608:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7147.191564] LustreError: 267608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7147.260871] Lustre: server umount lustre-MDT0000 complete [ 7150.281546] LustreError: 263384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350095 with bad export cookie 5257721290990416903 [ 7150.286674] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7150.293377] LustreError: 263384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7150.576566] Lustre: server umount lustre-MDT0001 complete [ 7158.934611] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7161.332879] Key type lgssc unregistered [ 7161.635883] LNet: 268340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7161.641684] LNetError: 268340:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7161.660125] LNet: Removed LNI 192.168.201.156@tcp [ 7162.284938] Key type .llcrypt unregistered [ 7162.287830] Key type ._llcrypt unregistered [ 7180.559427] Key type ._llcrypt registered [ 7180.561082] Key type .llcrypt registered [ 7180.664044] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7181.408284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7181.421081] alg: No test for adler32 (adler32-zlib) [ 7182.468845] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 7182.743684] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7184.385124] Key type lgssc registered [ 7185.296336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7191.065614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7191.083559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7192.617662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.787987] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7201.831219] LustreError: 269717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.851572] LustreError: 269717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7203.236416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7203.615125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7206.279857] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7210.697906] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7213.846193] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7219.302347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7219.626074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7224.067723] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7228.914947] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7237.034395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7262.694827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7262.698237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.727611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7265.493706] LustreError: 272757:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7265.569873] Lustre: server umount lustre-OST0000 complete [ 7270.369513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7270.374263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.384089] Lustre: Skipped 1 previous similar message [ 7270.392481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7270.395342] Lustre: Skipped 1 previous similar message [ 7272.930937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7272.932945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.691266] LustreError: 272959:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7274.724150] LustreError: 272959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7274.906945] Lustre: server umount lustre-MDT0000 complete [ 7278.049304] LustreError: 269717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.094601] LustreError: 269717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7278.491829] LustreError: 269697:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350223 with bad export cookie 7633595271873720933 [ 7278.518393] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7278.721258] LustreError: 273160:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7278.743731] LustreError: 273160:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7278.932171] Lustre: server umount lustre-MDT0001 complete [ 7288.854833] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7291.934427] Key type lgssc unregistered [ 7292.201942] LNet: 273692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7292.211282] LNetError: 273692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7292.224484] LNet: Removed LNI 192.168.201.156@tcp [ 7292.977622] Key type .llcrypt unregistered [ 7292.979157] Key type ._llcrypt unregistered [ 7305.771028] Key type ._llcrypt registered [ 7305.772765] Key type .llcrypt registered [ 7305.878177] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7306.692092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7306.762053] alg: No test for adler32 (adler32-zlib) [ 7307.744126] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 7307.960430] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7309.617402] Key type lgssc registered [ 7310.673296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7318.371760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7318.411245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.989656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7323.806618] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7330.275615] LustreError: 274875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7330.283766] LustreError: 274875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7331.282508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7331.665802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7335.057689] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7340.084893] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7343.213825] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7349.141686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7349.653329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7356.167741] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7362.040255] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7365.114985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7380.453723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7380.465920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7380.477379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7382.672171] LustreError: 277267:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7382.766298] Lustre: server umount lustre-OST0000 complete [ 7388.129516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7388.133507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7388.139049] Lustre: Skipped 1 previous similar message [ 7388.145295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7388.151614] Lustre: Skipped 1 previous similar message [ 7390.689326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7390.692032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.101639] LustreError: 277469:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7392.109125] LustreError: 277469:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7392.158386] Lustre: server umount lustre-MDT0000 complete [ 7395.275705] LustreError: 275594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350340 with bad export cookie 14844310349673303997 [ 7395.278920] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7395.283494] LustreError: 275594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7395.383841] LustreError: 277670:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7395.387218] LustreError: 277670:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7395.483969] Lustre: server umount lustre-MDT0001 complete [ 7403.496985] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7405.513344] Key type lgssc unregistered [ 7405.759598] LNet: 278203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7405.770347] LNetError: 278203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7405.786775] LNet: Removed LNI 192.168.201.156@tcp [ 7406.414827] Key type .llcrypt unregistered [ 7406.416815] Key type ._llcrypt unregistered [ 7424.610552] Key type ._llcrypt registered [ 7424.612254] Key type .llcrypt registered [ 7424.711302] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7425.507393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7425.533646] alg: No test for adler32 (adler32-zlib) [ 7426.515840] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 7426.709882] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7428.368199] Key type lgssc registered [ 7429.426503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7435.792729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7435.810672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.186738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.890883] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7447.526753] LustreError: 279581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7447.549941] LustreError: 279581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7447.619826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7447.957704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7451.051457] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7455.458623] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7458.706984] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7464.197802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7464.493547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7469.681637] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7474.864844] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7481.939994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7507.427154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7507.440589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7507.454947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7507.496906] Lustre: Skipped 1 previous similar message [ 7512.551531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.554238] Lustre: Skipped 1 previous similar message [ 7513.351083] LustreError: 282622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7513.489159] Lustre: server umount lustre-OST0000 complete [ 7517.667170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.682532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7517.695499] Lustre: Skipped 2 previous similar messages [ 7522.787772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7522.800630] Lustre: Skipped 1 previous similar message [ 7523.085568] LustreError: 282823:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7523.088542] LustreError: 282823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7523.218858] Lustre: server umount lustre-MDT0000 complete [ 7526.310530] LustreError: 279562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350471 with bad export cookie 2341641610642961573 [ 7526.316097] LustreError: 279562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7526.316127] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7526.436699] LustreError: 283024:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7526.447824] LustreError: 283024:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7526.582387] Lustre: server umount lustre-MDT0001 complete [ 7536.072373] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7537.861155] Key type lgssc unregistered [ 7538.136666] LNet: 283557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7538.146907] LNetError: 283557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7538.161181] LNet: Removed LNI 192.168.201.156@tcp [ 7538.854631] Key type .llcrypt unregistered [ 7538.856710] Key type ._llcrypt unregistered [ 7551.734850] Key type ._llcrypt registered [ 7551.736882] Key type .llcrypt registered [ 7551.813795] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7552.438364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7552.448225] alg: No test for adler32 (adler32-zlib) [ 7553.428956] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 7553.625209] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7555.264197] Key type lgssc registered [ 7556.128738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7562.838424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7562.854421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7564.431363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.829369] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7569.892843] LustreError: 284767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.911124] LustreError: 284767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7573.987993] LustreError: 284767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7575.064941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7575.432137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7578.664554] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7583.161883] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7585.988695] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7591.074405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7591.380651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7596.136177] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7600.366966] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7608.827951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7621.089597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7621.100811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7621.114467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7624.162141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7624.177519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7624.179400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.194512] Lustre: Skipped 1 previous similar message [ 7625.920436] LustreError: 287186:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7626.014828] Lustre: server umount lustre-OST0000 complete [ 7629.282577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.289172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7629.294037] Lustre: Skipped 1 previous similar message [ 7629.320151] Lustre: Skipped 1 previous similar message [ 7634.402815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7635.391964] LustreError: 287388:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7635.398725] LustreError: 287388:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7635.531855] Lustre: server umount lustre-MDT0000 complete [ 7638.651276] LustreError: 284721:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350583 with bad export cookie 13300954565767615964 [ 7638.674054] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.756123] LustreError: 287589:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7638.759788] LustreError: 287589:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7638.918318] Lustre: server umount lustre-MDT0001 complete [ 7647.359119] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7649.685759] Key type lgssc unregistered [ 7649.913861] LNet: 288121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7649.930131] LNetError: 288121:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7649.943991] LNet: Removed LNI 192.168.201.156@tcp [ 7650.609188] Key type .llcrypt unregistered [ 7650.611233] Key type ._llcrypt unregistered [ 7661.242980] Key type ._llcrypt registered [ 7661.244896] Key type .llcrypt registered [ 7661.345775] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7677.161887] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 7691.975564] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7692.840306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7692.885137] alg: No test for adler32 (adler32-zlib) [ 7694.393810] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 7694.617663] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7696.312151] Key type lgssc registered [ 7697.834646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7708.654056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7715.441490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7721.745748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7727.852389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7734.144719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7734.189771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7734.205728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.380977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7735.415422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7735.472888] Lustre: lustre-MDT0000: new disk, initializing [ 7735.565705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7735.594623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7738.906743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7747.856543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7747.932486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7748.018747] Lustre: 291631:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7748.078992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7748.081569] Lustre: Skipped 1 previous similar message [ 7748.150481] Lustre: lustre-MDT0001: new disk, initializing [ 7748.228324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7748.262307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7748.272883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7751.465645] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7758.169743] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7761.370635] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7767.295568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7767.378522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7767.669917] Lustre: lustre-OST0000: new disk, initializing [ 7767.676606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7767.774673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.811323] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7777.867351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7777.881186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7777.962404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7780.059724] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7784.348918] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7784.584253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7788.716490] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7788.906438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7793.634256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7793.637667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.653189] LustreError: Skipped 1 previous similar message [ 7793.665277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7793.671338] Lustre: Skipped 1 previous similar message [ 7797.084222] LustreError: 294570:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7797.226580] Lustre: server umount lustre-OST0000 complete [ 7803.876383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7803.883159] Lustre: Skipped 2 previous similar messages [ 7803.883255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7803.893410] Lustre: Skipped 1 previous similar message [ 7807.575473] LustreError: 294772:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7807.579679] LustreError: 294772:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7807.680926] Lustre: server umount lustre-MDT0000 complete [ 7811.196754] LustreError: 291622:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350756 with bad export cookie 8534005141487291481 [ 7811.202125] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7811.396966] LustreError: 294974:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7811.409926] LustreError: 294974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7811.621712] Lustre: server umount lustre-MDT0001 complete [ 7821.066843] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:39:24 (1763350764) [ 7827.152697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7827.530972] LustreError: 295645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.548933] LustreError: 295645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7827.594777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7831.078444] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7833.060328] LustreError: 295645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.170873] LustreError: 295645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.425104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7839.688138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7842.994743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7848.098114] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7851.332764] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7858.156809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7858.536155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7864.117207] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7869.657817] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7885.287500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7885.294066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.304163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7887.331514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7887.334395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7887.335312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7890.757108] LustreError: 297913:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7890.759943] LustreError: 297913:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7890.884112] Lustre: server umount lustre-OST0000 complete [ 7896.032957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7896.035646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7896.106232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.109053] Lustre: Skipped 1 previous similar message [ 7897.490660] LustreError: 298115:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7897.543420] LustreError: 298115:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7897.620117] LustreError: 295644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.700980] Lustre: server umount lustre-MDT0000 complete [ 7901.253525] LustreError: 295624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350846 with bad export cookie 8534005141487292363 [ 7901.265724] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7901.272414] LustreError: 295624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7901.619906] Lustre: server umount lustre-MDT0001 complete [ 7912.351742] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7914.849739] Key type lgssc unregistered [ 7915.106519] LNet: 298848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7915.111907] LNetError: 298848:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7915.124805] LNet: Removed LNI 192.168.201.156@tcp [ 7915.897541] Key type .llcrypt unregistered [ 7915.903813] Key type ._llcrypt unregistered [ 7943.225617] Key type ._llcrypt registered [ 7943.227769] Key type .llcrypt registered [ 7943.361305] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 7958.855139] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7960.358513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7960.518152] alg: No test for adler32 (adler32-zlib) [ 7961.674977] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 7961.898232] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7963.576399] Key type lgssc registered [ 7965.129282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7973.900935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7981.122874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7987.199854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7993.590105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7998.593357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8003.791552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8010.230562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8010.290699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8010.310130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8011.539318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8011.574355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8011.644657] Lustre: lustre-MDT0000: new disk, initializing [ 8011.721575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8011.739406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8015.747141] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8028.225779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8028.307362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8028.399420] Lustre: 302656:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8028.471578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8028.474251] Lustre: Skipped 1 previous similar message [ 8028.548972] Lustre: lustre-MDT0001: new disk, initializing [ 8028.687267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8028.772284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8028.788760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8032.345960] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8039.930487] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8043.853432] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8051.224728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8051.297301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8051.537181] Lustre: lustre-OST2710: new disk, initializing [ 8051.546908] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8051.634541] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8056.988559] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8059.982865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8059.995370] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8060.104044] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8064.267605] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8071.108445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8071.172368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8071.284100] Lustre: lustre-OST03e8: new disk, initializing [ 8071.301512] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8071.428591] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8077.393900] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8081.004263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8081.009950] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8081.185401] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8084.061353] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8090.472332] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8090.707858] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8094.579843] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8094.806882] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8098.785551] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8098.963733] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8103.081353] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8103.294870] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8106.988521] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.995409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.170803] LustreError: 307003:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8109.255717] Lustre: server umount lustre-MDT0000 complete [ 8112.581154] LustreError: 302648:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351057 with bad export cookie 13510298738708392809 [ 8112.584143] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8112.592193] LustreError: 302648:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8112.699066] LustreError: 307204:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8112.702895] LustreError: 307204:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8112.876827] Lustre: server umount lustre-MDT0001 complete [ 8126.487708] LustreError: 307405:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8126.495130] LustreError: 307405:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8126.582518] Lustre: server umount lustre-OST2710 complete [ 8140.245111] LustreError: 307608:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8140.250181] LustreError: 307608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8140.394979] Lustre: server umount lustre-OST03e8 complete [ 8158.055336] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 8167.407729] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8176.591074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8181.927422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8187.587935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8193.886691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8200.171477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8200.226717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8200.397971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8200.439548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8200.515196] Lustre: lustre-MDT0000: new disk, initializing [ 8200.618323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8200.633177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8204.667733] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8215.348503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8215.429305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8215.525682] Lustre: 310404:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8215.635446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8215.644764] Lustre: Skipped 1 previous similar message [ 8215.696825] Lustre: lustre-MDT0001: new disk, initializing [ 8215.788117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8215.838542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8215.884889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8220.526162] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8229.448720] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8233.679526] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8240.401333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8240.470473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8240.739698] Lustre: lustre-OST0000: new disk, initializing [ 8240.742730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8240.824796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8245.880325] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8247.363451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8247.375943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8247.524597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8253.564048] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8256.923578] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8257.130979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8260.348083] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8260.530242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8263.136990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8263.138708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.146197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8263.157045] Lustre: Skipped 4 previous similar messages [ 8263.159475] Lustre: Skipped 3 previous similar messages [ 8268.091664] LustreError: 313343:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8268.099672] LustreError: 313343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8268.264199] Lustre: server umount lustre-OST0000 complete [ 8272.353288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8272.363931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.374200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8277.542767] LustreError: 313545:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8277.545965] LustreError: 313545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8277.624192] Lustre: server umount lustre-MDT0000 complete [ 8278.498090] LustreError: 310417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.511279] LustreError: 310417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8280.856650] LustreError: 310396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351226 with bad export cookie 13510298738708394489 [ 8280.861343] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8280.869620] LustreError: 310396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8281.227744] Lustre: server umount lustre-MDT0001 complete [ 8289.902280] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:47:14 (1763351234) [ 8291.400126] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8293.093101] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:47:17 (1763351237) [ 8295.818833] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8302.996150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8307.501216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8311.917735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8316.850541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8318.798785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8326.910383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8327.227619] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8327.253251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8327.386219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8331.183015] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8338.843533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8338.917235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8338.939200] Lustre: 315881:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8343.324588] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8349.619755] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8354.431993] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8362.294493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8362.431215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8362.729708] Lustre: Denying initial registration attempt from nid 192.168.201.156@tcp, specified as failover [ 8362.737541] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8362.743766] LustreError: 317500:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 8362.756661] LustreError: 317500:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8362.759614] LustreError: 317500:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8362.822272] Lustre: server umount lustre-OST0000 complete [ 8362.829950] LustreError: 317500:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8370.151031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8370.158529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.188568] Lustre: Skipped 1 previous similar message [ 8370.196710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.204195] Lustre: Skipped 2 previous similar messages [ 8375.491214] LustreError: 317868:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8375.506594] LustreError: 317868:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8375.613298] Lustre: server umount lustre-MDT0000 complete [ 8376.812333] LustreError: 315894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.835242] LustreError: 315894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8379.144302] LustreError: 315874:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351324 with bad export cookie 13510298738708395371 [ 8379.150518] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8379.156620] LustreError: 315874:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8379.392089] Lustre: server umount lustre-MDT0001 complete [ 8389.697873] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8392.122497] Key type lgssc unregistered [ 8392.425907] LNet: 318605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8392.440182] LNetError: 318605:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8392.470388] LNet: Removed LNI 192.168.201.156@tcp [ 8393.320167] Key type .llcrypt unregistered [ 8393.323230] Key type ._llcrypt unregistered [ 8402.575363] Key type ._llcrypt registered [ 8402.576796] Key type .llcrypt registered [ 8402.668138] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8403.434403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8403.454104] alg: No test for adler32 (adler32-zlib) [ 8404.537391] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 8404.769881] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8406.408187] Key type lgssc registered [ 8407.424450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8413.812202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8416.905481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8420.291124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8423.832971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8424.783848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8439.494050] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8447.743815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8447.831947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.193569] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8449.212486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8449.398491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8453.486701] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8462.701344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8462.858477] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8462.889526] Lustre: 321379:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8463.154638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.299674] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8473.160475] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8476.856836] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8482.974988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8483.086579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8483.342590] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8483.536125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8489.125538] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8493.935484] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8497.636351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8497.637910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.642189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8497.646732] Lustre: Skipped 1 previous similar message [ 8497.652636] Lustre: Skipped 1 previous similar message [ 8502.219846] LustreError: 323667:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8502.323430] Lustre: server umount lustre-OST0000 complete [ 8507.881021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.887975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8512.421326] LustreError: 323868:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8512.425239] LustreError: 323868:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8512.555145] Lustre: server umount lustre-MDT0000 complete [ 8516.283844] LustreError: 321373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351461 with bad export cookie 3562846389169699882 [ 8516.289486] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8516.469524] LustreError: 324071:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8516.481022] LustreError: 324071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8516.694100] Lustre: server umount lustre-MDT0001 complete [ 8527.081270] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8529.416648] Key type lgssc unregistered [ 8529.646592] LNet: 324601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8529.659671] LNetError: 324601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8529.673410] LNet: Removed LNI 192.168.201.156@tcp [ 8530.429163] Key type .llcrypt unregistered [ 8530.433229] Key type ._llcrypt unregistered [ 8551.881670] Key type ._llcrypt registered [ 8551.883673] Key type .llcrypt registered [ 8551.986990] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8552.986591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8552.994944] alg: No test for adler32 (adler32-zlib) [ 8553.958201] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 8554.104184] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8555.744237] Key type lgssc registered [ 8556.643913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8562.939976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8562.962621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8564.556711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8568.581582] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8573.928791] LustreError: 325989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8573.942679] LustreError: 325989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8576.895855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8577.176141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8580.387670] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8585.623283] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8589.368310] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8595.118447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8595.337074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.234338] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8605.690106] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8622.926319] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 8623.077200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8623.090365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.100273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.101195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8624.139949] Lustre: Skipped 3 previous similar messages [ 8625.658360] LustreError: 328265:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8625.782693] Lustre: server umount lustre-MDT0000 complete [ 8628.022768] LustreError: 325986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8629.108042] LustreError: 325969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351574 with bad export cookie 17338364521300588056 [ 8629.112306] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8629.128116] LustreError: 325969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8629.217268] LustreError: 325986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8629.218804] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.238268] Lustre: Skipped 1 previous similar message [ 8629.241746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8633.142947] LustreError: 325984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8633.187743] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.56@tcp (stopping) [ 8635.280348] LustreError: 328466:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8635.293425] LustreError: 328466:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8635.417229] Lustre: server umount lustre-MDT0001 complete [ 8641.401265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8650.655569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.864340] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf09e3eab665b93f4 [ 8653.889359] Lustre: MGC192.168.201.156@tcp: Connection restored to 0@lo (at 0@lo) [ 8654.122233] LustreError: 327598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8654.132366] LustreError: 327598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8654.194290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8654.248099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8658.769826] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8664.422668] LustreError: 329274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8664.424544] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8664.437901] LustreError: 329274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8667.933353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8668.377762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8672.183207] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8673.786655] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8677.019542] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8679.514897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 8682.878934] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8684.318870] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8689.121485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8689.131609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.134205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8689.141650] Lustre: Skipped 1 previous similar message [ 8689.146590] Lustre: Skipped 1 previous similar message [ 8693.969538] LustreError: 330775:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8693.972308] LustreError: 330775:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8694.121199] Lustre: server umount lustre-OST0000 complete [ 8699.369820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.372250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8699.375925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8699.381128] Lustre: Skipped 1 previous similar message [ 8703.570185] LustreError: 330978:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8703.573947] LustreError: 330978:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8703.654324] Lustre: server umount lustre-MDT0000 complete [ 8704.483914] LustreError: 329275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.505993] LustreError: 329275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8706.620386] LustreError: 325970:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351651 with bad export cookie 17338364521300595700 [ 8706.627439] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8706.637846] LustreError: 325970:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8707.020721] Lustre: server umount lustre-MDT0001 complete [ 8716.187835] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8718.355932] Key type lgssc unregistered [ 8718.595765] LNet: 331713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8718.604119] LNetError: 331713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8718.617025] LNet: Removed LNI 192.168.201.156@tcp [ 8719.158916] Key type .llcrypt unregistered [ 8719.160713] Key type ._llcrypt unregistered [ 8738.735813] Key type ._llcrypt registered [ 8738.738157] Key type .llcrypt registered [ 8738.830194] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8739.678807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8739.702400] alg: No test for adler32 (adler32-zlib) [ 8740.688347] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 8740.866204] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8742.520327] Key type lgssc registered [ 8743.400817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8749.541781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8749.564590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.913279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8754.106693] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8761.206513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8761.315489] LustreError: 333089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.327117] LustreError: 333089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8761.489866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8764.489558] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8769.104723] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8772.010539] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8774.548770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.563269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8774.567471] Lustre: Skipped 1 previous similar message [ 8779.750719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.739741] LustreError: 334480:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8780.835302] Lustre: server umount lustre-MDT0000 complete [ 8784.256046] LustreError: 333070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351729 with bad export cookie 10054793629618401698 [ 8784.262976] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8784.282453] LustreError: 333070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8784.435957] LustreError: 334682:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8784.457919] LustreError: 334682:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8784.681901] Lustre: server umount lustre-MDT0001 complete [ 8796.586562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8796.856936] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8796.883057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8797.027407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8799.960440] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8806.134746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8806.201851] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8806.232331] Lustre: 335453:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8806.280235] LustreError: 335464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8806.396246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8808.976794] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8813.490942] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8816.460620] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8822.027669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8822.190876] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8822.206386] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8822.214655] LustreError: 337113:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 8822.218761] LustreError: 337113:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8822.227071] LustreError: 337113:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8822.287766] Lustre: server umount lustre-OST0000 complete [ 8822.290305] LustreError: 337113:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8829.161141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8829.281466] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8829.430848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8833.898760] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8836.204611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8838.341725] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8844.826743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8844.887225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8844.959404] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8845.037822] Lustre: lustre-OST0001: new disk, initializing [ 8845.040863] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8845.142795] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8849.324648] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8850.521536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8850.530486] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8850.587803] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8853.759636] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8861.161843] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8861.164250] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.165439] LustreError: Skipped 1 previous similar message [ 8861.173702] Lustre: Skipped 2 previous similar messages [ 8861.192989] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8861.198454] Lustre: Skipped 2 previous similar messages [ 8866.277436] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8866.282295] Lustre: Skipped 1 previous similar message [ 8870.372366] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8870.403779] LustreError: 339232:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8870.408059] LustreError: 339232:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8870.452229] Lustre: server umount lustre-OST0001 complete [ 8871.399410] LustreError: 338067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8871.420970] LustreError: 338067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8876.516039] LustreError: 337583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8876.516231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8876.518437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8876.527134] LustreError: 337583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8876.542122] Lustre: Skipped 1 previous similar message [ 8881.638776] LustreError: 337582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8881.647526] LustreError: 337582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8886.760831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.764014] Lustre: Skipped 4 previous similar messages [ 8887.264146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8887.307218] LustreError: 339434:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8887.315618] LustreError: 339434:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8887.404777] Lustre: server umount lustre-OST0000 complete [ 8891.876087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.882778] Lustre: Skipped 1 previous similar message [ 8896.095075] LustreError: 339637:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8896.099754] LustreError: 339637:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8896.159700] Lustre: server umount lustre-MDT0000 complete [ 8896.995159] LustreError: 336568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8897.001899] LustreError: 336568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8898.710838] LustreError: 335445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351844 with bad export cookie 10054793629618402251 [ 8898.716966] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8898.723026] LustreError: 335445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8898.922387] Lustre: server umount lustre-MDT0001 complete [ 8905.621562] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8907.412293] Key type lgssc unregistered [ 8907.598636] LNet: 340370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8907.613689] LNetError: 340370:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8908.650984] LNet: Removed LNI 192.168.201.156@tcp [ 8909.278848] Key type .llcrypt unregistered [ 8909.282883] Key type ._llcrypt unregistered [ 8931.319434] Key type ._llcrypt registered [ 8931.321353] Key type .llcrypt registered [ 8931.385753] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 8942.233975] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8942.801426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8942.903480] alg: No test for adler32 (adler32-zlib) [ 8943.921083] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 8944.074782] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8945.728162] Key type lgssc registered [ 8946.511292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8952.697957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8957.303396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8961.155949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8965.282800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8970.006100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8970.052886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8970.068083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8971.200347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8971.235330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8971.304859] Lustre: lustre-MDT0000: new disk, initializing [ 8971.362610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8971.377633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8973.851530] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8981.389281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8981.443612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8981.504246] Lustre: 344296:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8981.546883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8981.549257] Lustre: Skipped 1 previous similar message [ 8981.591070] Lustre: lustre-MDT0001: new disk, initializing [ 8981.662040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8981.687320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8981.693839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8984.063554] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8989.250649] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8991.317486] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8995.425128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8995.480270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8995.643449] Lustre: lustre-OST0000: new disk, initializing [ 8995.647388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8995.718488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8999.099541] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9002.544362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9002.549664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9002.651280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9003.961727] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9006.856873] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9006.989519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9009.374181] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9009.480972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9012.706300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9012.709985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9012.718028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9013.218960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9013.222835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9013.225713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9013.239599] Lustre: Skipped 1 previous similar message [ 9016.572154] LustreError: 347237:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9016.666637] Lustre: server umount lustre-OST0000 complete [ 9022.945912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9022.954452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9022.964056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9028.578235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9028.581264] Lustre: Skipped 3 previous similar messages [ 9033.184172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9033.267608] LustreError: 347438:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9033.270750] LustreError: 347438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9033.375800] Lustre: server umount lustre-MDT0000 complete [ 9035.510427] LustreError: 344289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351980 with bad export cookie 3537904359989609268 [ 9035.514659] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9035.515640] LustreError: 344289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9035.601981] LustreError: 347640:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9035.605768] LustreError: 347640:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9035.687649] Lustre: server umount lustre-MDT0001 complete [ 9040.007490] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 22:59:44 (1763351984) [ 9042.895716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9045.772849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9058.909808] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 9064.355966] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9070.151466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9074.447625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9078.095818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9081.534408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9085.432981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9085.473646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.599750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9085.625129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9085.680167] Lustre: lustre-MDT0000: new disk, initializing [ 9085.748562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9085.766296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9088.208424] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9095.303199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9095.347171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9095.402652] Lustre: 351409:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9095.439111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9095.442490] Lustre: Skipped 1 previous similar message [ 9095.507022] Lustre: lustre-MDT0001: new disk, initializing [ 9095.583795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9095.626248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9095.633473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9098.092231] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9103.067037] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9105.401747] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9109.908642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9109.973947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9110.129313] Lustre: lustre-OST0000: new disk, initializing [ 9110.133185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9110.197962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9113.738531] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9118.786505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9118.795885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9118.876772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9118.897402] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9121.459335] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9121.627680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9124.124746] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9124.274219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9129.448733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9129.450354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9129.450916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9129.454228] Lustre: Skipped 2 previous similar messages [ 9131.575241] LustreError: 354360:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9131.577419] LustreError: 354360:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9131.654124] Lustre: server umount lustre-OST0000 complete [ 9134.564060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9134.579172] Lustre: Skipped 1 previous similar message [ 9139.680954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9139.683981] Lustre: Skipped 4 previous similar messages [ 9148.384308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9148.443065] LustreError: 354562:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9148.445965] LustreError: 354562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9148.492799] Lustre: server umount lustre-MDT0000 complete [ 9148.898753] LustreError: 353240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9148.905980] LustreError: 353240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9150.736831] LustreError: 351402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352096 with bad export cookie 3537904359989610150 [ 9150.739312] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9150.746452] LustreError: 351402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9150.932435] Lustre: server umount lustre-MDT0001 complete [ 9155.850614] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:01:40 (1763352100) [ 9160.044132] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:01:44 (1763352104) [ 9165.270677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.533409] LustreError: 355777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9165.593029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9167.801279] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9170.913252] LustreError: 355777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9173.077678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9175.506660] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9178.963522] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9181.251250] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9185.804584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9189.416637] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9192.849644] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9202.295121] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9203.331964] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9204.373496] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9205.476868] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9206.638512] Lustre: Failing over lustre-MDT0000 [ 9206.796850] LustreError: 358216:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9206.801790] LustreError: 358216:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9206.886689] Lustre: server umount lustre-MDT0000 complete [ 9207.624160] LustreError: 355771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9208.803448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.811127] Lustre: Skipped 2 previous similar messages [ 9212.733596] LustreError: 357934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.743541] LustreError: 357934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9222.692094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9222.781414] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.955685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9222.958696] Lustre: Skipped 2 previous similar messages [ 9222.997948] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9225.699115] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9228.084775] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9228.267553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9228.283610] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9228.307418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9230.419454] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9231.484770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9234.009564] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9235.233476] Lustre: Failing over lustre-MDT0000 [ 9235.359400] LustreError: 359296:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9235.363612] LustreError: 359296:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9235.426334] Lustre: server umount lustre-MDT0000 complete [ 9238.348583] LustreError: 355773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9238.361256] LustreError: 355773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9238.500628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.503131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9238.507097] Lustre: Skipped 2 previous similar messages [ 9250.317648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9250.427903] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9250.620813] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9253.192681] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9253.685634] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9255.910351] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9255.916513] Lustre: Skipped 2 previous similar messages [ 9255.934541] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9255.967201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9258.208226] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9259.268269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9266.152154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9266.156450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9266.166425] Lustre: Skipped 4 previous similar messages [ 9268.833155] LustreError: 360325:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9268.836606] LustreError: 360325:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9268.849744] LustreError: 360325:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000000d1d3979) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 9270.907108] Lustre: server umount lustre-MDT0000 complete [ 9271.266970] LustreError: 355776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9271.277216] LustreError: 355776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 9272.925310] LustreError: 355757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352218 with bad export cookie 3537904359989612845 [ 9272.929273] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9272.930333] LustreError: 355757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9273.111560] Lustre: server umount lustre-MDT0001 complete [ 9285.593933] Lustre: server umount lustre-OST0000 complete [ 9296.243884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9296.580262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9296.583093] Lustre: Skipped 1 previous similar message [ 9298.988459] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9304.327851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9306.688942] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9309.982231] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9312.335291] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9317.073841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9320.510442] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9321.387719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9323.837846] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9336.801429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9336.805524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9336.811404] Lustre: Skipped 3 previous similar messages [ 9336.815203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9342.838124] LustreError: 363826:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9342.846782] LustreError: 363826:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9342.921468] Lustre: server umount lustre-OST0000 complete [ 9351.366452] Lustre: server umount lustre-MDT0000 complete [ 9353.346910] LustreError: 361536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352298 with bad export cookie 3537904359989613580 [ 9353.353177] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9353.356035] LustreError: 361536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9353.523618] Lustre: server umount lustre-MDT0001 complete [ 9358.588669] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 9360.028945] Key type lgssc unregistered [ 9360.181885] LNet: 364760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9360.185874] LNetError: 364760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9360.194519] LNet: Removed LNI 192.168.201.156@tcp [ 9360.640184] Key type .llcrypt unregistered [ 9360.642069] Key type ._llcrypt unregistered [ 9375.416927] Key type ._llcrypt registered [ 9375.418780] Key type .llcrypt registered [ 9375.480826] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9375.882646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9375.898434] alg: No test for adler32 (adler32-zlib) [ 9376.812499] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 9376.929886] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 9378.536187] Key type lgssc registered [ 9379.204092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9383.812437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9383.831663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9385.106205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9387.306450] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9392.974855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9393.223440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9395.650590] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9399.185443] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9401.075392] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9404.751821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9404.938086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9407.951771] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9410.094550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9411.538288] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9428.451621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.453558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9428.457419] Lustre: Skipped 2 previous similar messages [ 9428.462888] Lustre: Skipped 2 previous similar messages [ 9433.106246] LustreError: 368503:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9433.166721] Lustre: server umount lustre-MDT0000 complete [ 9433.571063] LustreError: 367451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9433.577378] LustreError: 367451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9435.268489] LustreError: 366217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352380 with bad export cookie 12750301689844649605 [ 9435.272074] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9435.274409] LustreError: 366217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9435.367854] LustreError: 368704:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9435.370832] LustreError: 368704:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9435.466861] Lustre: server umount lustre-MDT0001 complete [ 9446.786940] LustreError: 368905:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9446.790944] LustreError: 368905:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9446.864130] Lustre: server umount lustre-OST0000 complete [ 9453.951947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9454.226956] LustreError: 369634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9454.267823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.458250] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9459.681287] LustreError: 369634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.583160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9461.794272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9464.077394] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9467.765347] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9469.974290] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9474.177056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9474.372245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9477.964507] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9479.535877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9481.359437] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9491.940279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.942256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.946263] Lustre: Skipped 2 previous similar messages [ 9491.948429] Lustre: Skipped 1 previous similar message [ 9495.952027] LustreError: 371980:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9495.955076] LustreError: 371980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9496.005815] Lustre: server umount lustre-MDT0000 complete [ 9497.058609] LustreError: 369658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.069082] LustreError: 369658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9498.090455] LustreError: 369613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352443 with bad export cookie 12750301689844650704 [ 9498.094294] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9498.098787] LustreError: 369613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9498.283994] Lustre: server umount lustre-MDT0001 complete [ 9510.909870] LustreError: 372383:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9510.913121] LustreError: 372383:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9511.013903] Lustre: server umount lustre-OST0000 complete [ 9520.755465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.044104] LustreError: 373211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9521.086577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9523.252159] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9527.928207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9530.325968] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9533.613173] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9535.622097] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9539.445199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9543.080904] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9545.785290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9546.149402] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9550.819669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9550.820079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.834933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9550.838184] Lustre: Skipped 1 previous similar message [ 9552.871594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9554.568330] LustreError: 375456:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9554.572384] LustreError: 375456:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9554.639153] Lustre: server umount lustre-OST0000 complete [ 9557.989065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.995450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9557.999938] Lustre: Skipped 1 previous similar message [ 9558.005019] Lustre: Skipped 1 previous similar message [ 9568.226194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9568.231512] Lustre: Skipped 4 previous similar messages [ 9571.296321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9571.355101] LustreError: 375658:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9571.357833] LustreError: 375658:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9571.413126] Lustre: server umount lustre-MDT0000 complete [ 9573.347065] LustreError: 373206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9573.355843] LustreError: 373206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9573.510759] LustreError: 373190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352518 with bad export cookie 12750301689844651719 [ 9573.512791] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9573.517567] LustreError: 373190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9573.701475] Lustre: server umount lustre-MDT0001 complete [ 9579.354746] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 9581.106071] Key type lgssc unregistered [ 9581.301481] LNet: 376391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9581.308265] LNetError: 376391:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9581.323253] LNet: Removed LNI 192.168.201.156@tcp [ 9581.821712] Key type .llcrypt unregistered [ 9581.823429] Key type ._llcrypt unregistered [ 9597.958780] Key type ._llcrypt registered [ 9597.960294] Key type .llcrypt registered [ 9598.041996] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9598.773068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9598.784632] alg: No test for adler32 (adler32-zlib) [ 9599.734636] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 9599.875127] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 9601.504170] Key type lgssc registered [ 9602.220973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9607.598102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9607.617933] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.627198] LustreError: 377790:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9607.651053] LustreError: 377790:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9607.655747] LustreError: 377790:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9607.660335] LustreError: 377790:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9607.667360] LustreError: 377790:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9612.586824] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9612.591701] LustreError: 378188:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9612.605374] LustreError: 378188:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9612.610314] LustreError: 378188:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9612.614733] LustreError: 378188:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9612.620151] LustreError: 378188:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9623.369139] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 9624.746098] Key type lgssc unregistered [ 9624.919699] LNet: 379172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9624.924088] LNetError: 379172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9624.932667] LNet: Removed LNI 192.168.201.156@tcp [ 9625.292598] Key type .llcrypt unregistered [ 9625.294204] Key type ._llcrypt unregistered [ 9634.690577] Key type ._llcrypt registered [ 9634.692046] Key type .llcrypt registered [ 9634.758479] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 9643.201878] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9643.615476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9643.661070] alg: No test for adler32 (adler32-zlib) [ 9644.585501] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 9644.715377] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 9646.328161] Key type lgssc registered [ 9646.929094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9651.385515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9654.619472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9658.043166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9661.381049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9665.324976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9665.359127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9665.368783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.507863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9666.531592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9666.588399] Lustre: lustre-MDT0000: new disk, initializing [ 9666.653207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.666901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9668.810423] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9674.958103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9674.996358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9675.065201] Lustre: 382281:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9675.101467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9675.105306] Lustre: Skipped 1 previous similar message [ 9675.148663] Lustre: lustre-MDT0001: new disk, initializing [ 9675.208584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9675.237407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9675.244727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9677.319331] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9681.770568] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9683.594694] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9687.215978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9687.262818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9687.393668] Lustre: lustre-OST0000: new disk, initializing [ 9687.396965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9687.445857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.786733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9688.793968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9688.873448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9690.465794] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9694.765785] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9696.927231] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9697.053309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9699.567577] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9699.696534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9704.423313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9704.425060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.430027] LustreError: Skipped 1 previous similar message [ 9704.447407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9704.450337] Lustre: Skipped 1 previous similar message [ 9709.537787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9709.551943] Lustre: Skipped 1 previous similar message [ 9714.661577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9714.666114] Lustre: Skipped 1 previous similar message [ 9715.168397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9715.196549] LustreError: 385220:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9715.258196] Lustre: server umount lustre-OST0000 complete [ 9719.780510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.787931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9719.789804] Lustre: Skipped 2 previous similar messages [ 9719.795479] Lustre: Skipped 1 previous similar message [ 9723.599606] LustreError: 385423:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9723.602269] LustreError: 385423:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9723.641796] Lustre: server umount lustre-MDT0000 complete [ 9725.841365] LustreError: 383062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352671 with bad export cookie 3275167093198224027 [ 9725.842693] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9725.846816] LustreError: 383062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9725.959758] LustreError: 385625:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9725.963060] LustreError: 385625:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9726.107847] Lustre: server umount lustre-MDT0001 complete [ 9731.392331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9732.406703] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:11:17 (1763352677) [ 9736.246251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.534669] LustreError: 386345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9736.547838] LustreError: 386345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9736.588144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9739.201551] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9741.794451] LustreError: 386345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9744.679890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9744.960501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9747.797321] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9751.782373] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9754.263408] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9758.665215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9758.848631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9762.145745] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9765.488892] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9769.670217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9769.717622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9769.830554] Lustre: lustre-OST0001: new disk, initializing [ 9769.837916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9770.422930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9770.430721] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9770.491540] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9773.227930] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9778.094551] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9780.705211] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9780.707457] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9780.708830] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9780.711903] Lustre: Skipped 1 previous similar message [ 9786.680659] LustreError: 389541:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9786.685584] LustreError: 389541:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9786.713446] Lustre: server umount lustre-OST0001 complete [ 9786.853844] LustreError: 388911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.861862] LustreError: 388911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9791.973136] LustreError: 389543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9791.982028] LustreError: 389543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9797.094719] LustreError: 388911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.102773] LustreError: 388911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9802.211293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9802.214851] LustreError: Skipped 1 previous similar message [ 9802.216549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9802.220644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9802.236142] Lustre: Skipped 1 previous similar message [ 9802.239168] Lustre: Skipped 4 previous similar messages [ 9806.010122] LustreError: 389767:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9806.016432] LustreError: 389767:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9806.102974] Lustre: server umount lustre-OST0000 complete [ 9811.429327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9811.438180] LustreError: Skipped 1 previous similar message [ 9811.440492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9814.756420] LustreError: 389969:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9814.761080] LustreError: 389969:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9814.800447] Lustre: server umount lustre-MDT0000 complete [ 9817.022402] LustreError: 386325:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352762 with bad export cookie 3275167093198224909 [ 9817.026507] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9817.034372] LustreError: 386325:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9831.392236] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9831.480940] LustreError: 390170:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9831.488429] LustreError: 390170:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9831.597823] Lustre: server umount lustre-MDT0001 complete [ 9837.579677] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 9839.092947] Key type lgssc unregistered [ 9839.248620] LNet: 390703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9839.253445] LNetError: 390703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9839.265921] LNet: Removed LNI 192.168.201.156@tcp [ 9839.703642] Key type .llcrypt unregistered [ 9839.705421] Key type ._llcrypt unregistered [ 9857.590175] Key type ._llcrypt registered [ 9857.591788] Key type .llcrypt registered [ 9857.651764] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 9865.619302] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9866.152758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9866.170199] alg: No test for adler32 (adler32-zlib) [ 9867.077580] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 9867.207338] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 9868.816316] Key type lgssc registered [ 9869.322696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9873.661527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9876.980755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9880.524758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9884.402476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9888.618225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9888.652228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9888.665204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.840129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9889.868991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9889.919660] Lustre: lustre-MDT0000: new disk, initializing [ 9889.973809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9889.986626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9892.227141] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9899.044833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9899.082672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9899.132720] Lustre: 394628:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9899.173831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9899.177167] Lustre: Skipped 1 previous similar message [ 9899.241099] Lustre: lustre-MDT0001: new disk, initializing [ 9899.317686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9899.345871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9899.353339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9901.698722] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9906.016382] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9908.074224] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9911.367785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9911.399097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9911.528689] Lustre: lustre-OST0000: new disk, initializing [ 9911.531286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9911.573287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9914.174651] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9918.410456] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9919.507136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9919.512543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9919.575457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9920.427980] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9920.523373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9922.442881] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9922.571739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9924.583284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9924.583594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9924.588152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9924.592796] Lustre: Skipped 1 previous similar message [ 9924.598294] Lustre: Skipped 1 previous similar message [ 9925.685124] LustreError: 397565:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9925.767032] Lustre: server umount lustre-OST0000 complete [ 9929.698937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9929.700310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9929.704993] Lustre: Skipped 1 previous similar message [ 9933.970672] LustreError: 397766:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9933.973169] LustreError: 397766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9934.016628] Lustre: server umount lustre-MDT0000 complete [ 9935.995336] LustreError: 395409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352881 with bad export cookie 11057771053902026155 [ 9935.997673] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9936.000880] LustreError: 395409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9936.088216] LustreError: 397968:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9936.090490] LustreError: 397968:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9936.183051] Lustre: server umount lustre-MDT0001 complete [ 9940.563769] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 23:14:45 (1763352885) [ 9944.827796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9949.140701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9949.346188] Lustre: lustre-MDT0000: new disk, initializing [ 9949.364840] LustreError: 399235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.373484] LustreError: 399235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9949.408073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9951.333341] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9954.786653] LustreError: 399235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9955.402830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9955.600146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9957.581829] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9960.711343] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9963.169552] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9965.537618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9965.539040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9965.544826] Lustre: Skipped 1 previous similar message [ 9965.550765] Lustre: Skipped 2 previous similar messages [ 9970.634913] LustreError: 400578:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9970.638178] LustreError: 400578:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9970.657720] LustreError: 399235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.718802] Lustre: server umount lustre-MDT0000 complete [ 9972.767843] LustreError: 399214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352918 with bad export cookie 11057771053902027023 [ 9972.770911] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9972.773289] LustreError: 399214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9972.943957] Lustre: server umount lustre-MDT0001 complete [ 9978.064107] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:15:22 (1763352922) [ 9981.735622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.972327] LustreError: 401499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9981.983904] LustreError: 401499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9982.012418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9983.985470] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9987.041887] LustreError: 401498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9987.048600] LustreError: 401498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9988.488742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9990.645101] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9993.462740] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9995.239935] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9998.500255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9998.692558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9998.695906] Lustre: Skipped 1 previous similar message [10001.653132] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10004.277963] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10013.493454] Lustre: Permanently deactivating lustre-OST0000 [10013.502327] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10016.826485] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10017.291139] LustreError: 404044:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10017.296555] LustreError: 404044:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10017.299643] LustreError: 404044:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10018.157911] LustreError: 404092:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10018.161326] LustreError: 404092:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10025.207246] LustreError: 404189:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10025.211663] LustreError: 404189:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10025.294188] Lustre: server umount lustre-OST0000 complete [10026.736657] LustreError: 404341:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10026.740338] LustreError: 404341:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10029.537508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10029.541181] LustreError: Skipped 1 previous similar message [10029.543121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10029.551968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10035.683814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10035.686618] Lustre: Skipped 3 previous similar messages [10037.738866] LustreError: 404438:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10037.741827] LustreError: 404438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10037.778604] Lustre: server umount lustre-MDT0000 complete [10039.471531] LustreError: 402217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352984 with bad export cookie 11057771053902027562 [10039.475400] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10039.478174] LustreError: 402217:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10049.077389] Lustre: server umount lustre-MDT0001 complete [10054.101310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.326667] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10057.675996] LustreError: 405485:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10060.974137] Lustre: 405823:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.156@tcp [10061.997495] LustreError: 405920:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10062.000328] LustreError: 405920:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10062.050782] Lustre: server umount MGS complete [10067.710037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10067.913642] LustreError: 406548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10067.919657] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10067.925538] Lustre: Skipped 1 previous similar message [10067.948782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10069.552616] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10073.470035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10073.598646] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10075.413325] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10077.763364] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10079.337935] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10080.201112] Lustre: Permanently reactivating lustre-OST0000 [10080.216164] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10080.219166] Lustre: Skipped 2 previous similar messages [10104.288433] LustreError: 408055:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10108.410796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10110.885403] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10113.229182] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10129.379843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10129.380750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10129.384749] LustreError: Skipped 1 previous similar message [10129.389022] Lustre: Skipped 1 previous similar message [10129.392819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10129.399452] Lustre: Skipped 1 previous similar message [10132.276722] LustreError: 409839:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10132.279460] LustreError: 409839:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10132.329616] Lustre: server umount lustre-OST0000 complete [10140.560318] Lustre: server umount lustre-MDT0000 complete [10142.411518] LustreError: 407898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353087 with bad export cookie 11057771053902028521 [10142.414903] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10142.416642] LustreError: 407898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10142.579203] Lustre: server umount lustre-MDT0001 complete [10147.899217] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10149.249295] Key type lgssc unregistered [10149.412986] LNet: 410774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10149.417245] LNetError: 410774:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10149.426965] LNet: Removed LNI 192.168.201.156@tcp [10149.828790] Key type .llcrypt unregistered [10149.830359] Key type ._llcrypt unregistered [10158.598134] Key type ._llcrypt registered [10158.599615] Key type .llcrypt registered [10158.657409] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [10166.211334] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10166.530840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10166.547709] alg: No test for adler32 (adler32-zlib) [10167.465226] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [10167.576709] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [10169.187541] Key type lgssc registered [10169.709327] Lustre: Echo OBD driver; http://www.lustre.org/ [10173.805097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10176.817793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10179.504585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10182.299833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10185.406634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10185.428770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10185.435604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10186.548913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10186.568166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10186.611145] Lustre: lustre-MDT0000: new disk, initializing [10186.654142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10186.665782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10188.362748] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10193.700843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10193.737119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10193.781909] Lustre: 413884:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10193.812886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10193.815737] Lustre: Skipped 1 previous similar message [10193.856631] Lustre: lustre-MDT0001: new disk, initializing [10193.904611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10193.926434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10193.937812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10195.827965] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10200.066088] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10201.827775] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10205.039079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10205.079814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10205.214697] Lustre: lustre-OST0000: new disk, initializing [10205.216861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10205.259151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10205.965894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10205.970846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10206.019822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10207.784442] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10211.431674] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10212.963224] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10213.050866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10214.568640] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10214.655158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10216.419327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10216.421258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10216.422680] LustreError: Skipped 1 previous similar message [10216.422977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10216.427979] Lustre: Skipped 1 previous similar message [10221.493119] LustreError: 416822:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10221.551730] Lustre: server umount lustre-OST0000 complete [10224.609463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10224.612752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10224.618903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10224.621446] Lustre: Skipped 1 previous similar message [10226.657132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10226.659162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10229.382332] LustreError: 417023:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10229.385201] LustreError: 417023:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10229.427154] Lustre: server umount lustre-MDT0000 complete [10231.322352] LustreError: 414663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353176 with bad export cookie 17982026043696195584 [10231.322703] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10231.329833] LustreError: 414663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10231.420569] LustreError: 417224:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10231.423628] LustreError: 417224:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10231.494731] Lustre: server umount lustre-MDT0001 complete [10235.353123] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10236.166484] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:19:41 (1763353181) [10239.518371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.802729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10241.940651] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10245.091593] LustreError: 417942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10245.105221] LustreError: 417942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10246.156887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10246.384193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10248.199475] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10250.577173] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10252.118337] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10254.998265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10255.163206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10257.787670] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10260.018414] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10260.853636] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10267.590147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10270.690809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.693373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10270.696534] Lustre: Skipped 1 previous similar message [10270.700762] Lustre: Skipped 1 previous similar message [10270.706928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10275.811093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10275.817417] Lustre: Skipped 2 previous similar messages [10276.090285] LustreError: 420380:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10276.096341] LustreError: 420380:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10276.159269] Lustre: server umount lustre-OST0000 complete [10280.931485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10284.360684] LustreError: 420581:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10284.364074] LustreError: 420581:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10284.413544] Lustre: server umount lustre-MDT0000 complete [10286.049581] LustreError: 420243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.056420] LustreError: 420243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10286.155324] LustreError: 417923:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353231 with bad export cookie 17982026043696196459 [10286.158705] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10286.161203] LustreError: 417923:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10286.325964] Lustre: server umount lustre-MDT0001 complete [10291.093630] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10292.343465] Key type lgssc unregistered [10292.489466] LNet: 421315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10292.493597] LNetError: 421315:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10292.502527] LNet: Removed LNI 192.168.201.156@tcp [10292.844101] Key type .llcrypt unregistered [10292.845594] Key type ._llcrypt unregistered [10304.028105] Key type ._llcrypt registered [10304.029889] Key type .llcrypt registered [10304.077572] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10304.211430] Key type .llcrypt unregistered [10304.213199] Key type ._llcrypt unregistered [10312.438569] Key type ._llcrypt registered [10312.440310] Key type .llcrypt registered [10312.488516] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10312.888475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10312.920050] alg: No test for adler32 (adler32-zlib) [10313.806489] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [10313.919849] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [10315.530542] Key type lgssc registered [10316.102369] Lustre: Echo OBD driver; http://www.lustre.org/ [10319.680388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10319.692726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10320.929316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10322.721611] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10326.560701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10326.728031] LustreError: 423462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.736179] LustreError: 423462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10326.737494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10329.278274] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10330.853365] LustreError: 423466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.860755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10330.869107] LustreError: 423466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10331.848285] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10334.861492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10335.030443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10336.664350] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10345.443364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.450727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10345.452622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10345.460587] Lustre: Skipped 1 previous similar message [10350.323130] LustreError: 425522:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10350.382780] Lustre: server umount lustre-OST0000 complete [10355.172349] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10355.176104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10355.185353] Lustre: Skipped 1 previous similar message [10355.189519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10358.279797] LustreError: 425724:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10358.283604] LustreError: 425724:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10358.330123] Lustre: server umount lustre-MDT0000 complete [10359.998362] LustreError: 423447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353305 with bad export cookie 6557993953459019074 [10360.002485] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10360.004189] LustreError: 423447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10360.076153] LustreError: 425926:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10360.078649] LustreError: 425926:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10360.156458] Lustre: server umount lustre-MDT0001 complete [10364.671376] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10365.921314] Key type lgssc unregistered [10366.059596] LNet: 426457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10366.062647] LNetError: 426457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10366.070608] LNet: Removed LNI 192.168.201.156@tcp [10366.412345] Key type .llcrypt unregistered [10366.414050] Key type ._llcrypt unregistered [10377.181990] Key type ._llcrypt registered [10377.183523] Key type .llcrypt registered [10377.231428] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10377.586272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10377.594684] alg: No test for adler32 (adler32-zlib) [10378.477936] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [10378.582911] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [10380.176195] Key type lgssc registered [10380.652345] Lustre: Echo OBD driver; http://www.lustre.org/ [10384.066868] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10384.082801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10399.584440] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10404.704283] LustreError: 427803:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [10431.328205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10433.731323] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10436.228355] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10439.427442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10439.673505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10439.713893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10441.468952] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10444.770791] LustreError: 427830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10444.777786] LustreError: 427830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10445.320290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10445.531597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10447.390471] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10450.141961] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10451.814490] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10455.010580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10455.015181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.023753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10456.033635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10456.037220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.043276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10457.058323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10457.060834] Lustre: Skipped 1 previous similar message [10460.655130] LustreError: 430083:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [10460.714756] Lustre: server umount lustre-OST0000 complete [10464.738973] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10464.742190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10464.748447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10467.298593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10468.376381] LustreError: 430285:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10468.380099] LustreError: 430285:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10468.424115] Lustre: server umount lustre-MDT0000 complete [10470.080918] LustreError: 429452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353415 with bad export cookie 11651228005993408482 [10470.082655] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10470.086258] LustreError: 429452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10470.170949] LustreError: 430486:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10470.173163] LustreError: 430486:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10470.241885] Lustre: server umount lustre-MDT0001 complete [10474.889875] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10476.264912] Key type lgssc unregistered [10476.427667] LNet: 431017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10476.431342] LNetError: 431017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10476.444505] LNet: Removed LNI 192.168.201.156@tcp [10476.853122] Key type .llcrypt unregistered [10476.855142] Key type ._llcrypt unregistered [10488.569102] Key type ._llcrypt registered [10488.570727] Key type .llcrypt registered [10488.632748] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10489.008858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10489.044041] alg: No test for adler32 (adler32-zlib) [10489.932130] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [10490.055421] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [10491.667760] Key type lgssc registered [10492.212364] Lustre: Echo OBD driver; http://www.lustre.org/ [10495.802104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10495.811733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10497.035118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10498.680508] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10502.245965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10502.403521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10504.039281] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10506.427925] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10508.050276] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10511.052869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10511.207720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10513.689413] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10516.157472] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10522.473568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10532.685656] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [10532.835560] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10532.841426] Lustre: Skipped 2 previous similar messages [10534.786207] LustreError: 434661:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10534.828254] Lustre: server umount lustre-MDT0000 complete [10535.959490] LustreError: 434663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.56@tcp arrived at 1763353481 with bad export cookie 7985274439189540157 [10535.967701] LustreError: 434663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10536.916096] LustreError: 434663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353482 with bad export cookie 7985274439189539506 [10536.921111] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10536.922590] LustreError: 434663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10537.954414] LustreError: 434195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10537.955549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.956740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10537.965288] LustreError: 434195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10537.980826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10537.984381] Lustre: Skipped 3 previous similar messages [10541.985132] LustreError: 434195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10541.985424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10541.994609] LustreError: 434195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10543.034264] LustreError: 434864:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10543.039139] LustreError: 434864:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10543.110787] Lustre: server umount lustre-OST0000 complete [10546.702978] LustreError: 435218:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10546.714843] LustreError: 435218:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10546.808819] Lustre: server umount lustre-MDT0001 complete [10551.570065] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10552.946831] Key type lgssc unregistered [10553.095876] LNet: 435750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10553.101613] LNetError: 435750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10553.110682] LNet: Removed LNI 192.168.201.156@tcp [10553.514986] Key type .llcrypt unregistered [10553.516612] Key type ._llcrypt unregistered [10565.013633] Key type ._llcrypt registered [10565.015918] Key type .llcrypt registered [10565.062861] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10565.602270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10565.653831] alg: No test for adler32 (adler32-zlib) [10566.559649] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [10566.669164] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [10568.264371] Key type lgssc registered [10568.769948] Lustre: Echo OBD driver; http://www.lustre.org/ [10572.307144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10572.315656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10573.549816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10575.285600] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10579.341036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10579.522749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10581.464632] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10584.224560] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10586.047289] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10589.118402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10589.266148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10591.678565] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10593.984858] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10598.503088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10604.003600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10604.004123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10604.010288] Lustre: Skipped 1 previous similar message [10605.364986] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.56@tcp (stopping) [10605.369273] Lustre: Skipped 1 previous similar message [10607.234683] LustreError: 439368:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10607.265839] Lustre: server umount lustre-MDT0001 complete [10609.122871] LustreError: 437125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.133823] LustreError: 437125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10614.242165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10614.243558] LustreError: 437125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10614.245185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10614.246156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10615.923154] LustreError: 439572:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10615.928639] LustreError: 439572:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10615.976607] Lustre: server umount lustre-OST0000 complete [10617.767374] LustreError: 439774:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10617.770631] LustreError: 439774:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10617.853078] Lustre: server umount lustre-MDT0000 complete [10623.623758] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10624.844936] Key type lgssc unregistered [10624.990744] LNet: 440457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10624.994227] LNetError: 440457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10625.003401] LNet: Removed LNI 192.168.201.156@tcp [10625.361850] Key type .llcrypt unregistered [10625.363245] Key type ._llcrypt unregistered [10635.303973] Key type ._llcrypt registered [10635.305641] Key type .llcrypt registered [10635.355652] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [10642.183458] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10642.519355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10642.529327] alg: No test for adler32 (adler32-zlib) [10643.419197] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [10643.530785] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [10645.128270] Key type lgssc registered [10645.664709] Lustre: Echo OBD driver; http://www.lustre.org/ [10649.656681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10652.557661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10655.484253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10658.586431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10662.679562] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10666.526580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10669.941398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10669.980908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10669.995570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10671.112189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10671.130878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10671.174901] Lustre: lustre-MDT0000: new disk, initializing [10671.230109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10671.246962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10673.024675] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10677.546672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10680.618910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10680.654328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10680.778670] Lustre: lustre-OST0000: new disk, initializing [10680.781234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10680.784159] Lustre: Skipped 1 previous similar message [10680.818414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10683.230550] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10685.467050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10685.475310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10685.493940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10687.067388] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10691.339889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10694.691396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10694.718683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10694.768311] Lustre: 444403:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10694.815085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10694.867308] Lustre: lustre-MDT0001: new disk, initializing [10694.924110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10694.947122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10694.955279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10696.757801] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10701.294605] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10701.401168] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10703.253431] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10704.399111] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [10708.963820] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10708.964187] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10708.970697] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10708.976469] Lustre: Skipped 1 previous similar message [10710.847992] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.56@tcp (stopping) [10712.642336] LustreError: 447832:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10712.676794] Lustre: server umount lustre-MDT0001 complete [10719.203542] LustreError: 444410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10719.210212] LustreError: 444410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10724.324759] LustreError: 444410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10724.332327] LustreError: 444410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10726.368958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10726.372209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10726.377399] Lustre: Skipped 1 previous similar message [10726.381163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10729.441731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10729.441775] LustreError: 447565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10731.760452] LustreError: 448105:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [10731.766788] LustreError: 448105:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10731.827269] Lustre: server umount lustre-OST0000 complete [10739.589619] LustreError: 448306:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [10739.593494] LustreError: 448306:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10739.664374] Lustre: server umount lustre-MDT0000 complete [10745.253286] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10746.478494] Key type lgssc unregistered [10746.645430] LNet: 448990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10746.649342] LNetError: 448990:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10746.664493] LNet: Removed LNI 192.168.201.156@tcp [10747.075902] Key type .llcrypt unregistered [10747.078468] Key type ._llcrypt unregistered [10766.172977] Key type ._llcrypt registered [10766.174284] Key type .llcrypt registered [10766.231476] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10766.377600] Key type .llcrypt unregistered [10766.379345] Key type ._llcrypt unregistered [10774.962149] Key type ._llcrypt registered [10774.963860] Key type .llcrypt registered [10775.011895] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10775.465862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10775.486902] alg: No test for adler32 (adler32-zlib) [10776.385151] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [10776.488556] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [10778.088171] Key type lgssc registered [10778.604602] Lustre: Echo OBD driver; http://www.lustre.org/ [10782.309449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10785.311886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10788.088669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10791.083268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10791.112231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10791.120270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.252369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10792.278064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10792.335580] Lustre: lustre-MDT0000: new disk, initializing [10792.380323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10792.392617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10794.337616] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10799.804278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10799.837080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10799.890698] Lustre: 452307:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10799.921900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10799.925354] Lustre: Skipped 1 previous similar message [10799.967824] Lustre: lustre-MDT0001: new disk, initializing [10800.017760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10800.034964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10800.039974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10801.710428] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10805.548425] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10807.182642] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10810.260869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10810.293291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10810.412674] Lustre: lustre-OST0000: new disk, initializing [10810.415410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10810.455341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10813.190758] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10814.535813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10814.539929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10814.594460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10817.110205] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10820.577270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10820.582630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10820.589263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.187054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.190081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.196784] Lustre: Skipped 1 previous similar message [10826.566039] LustreError: 454748:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10826.611938] Lustre: server umount lustre-MDT0000 complete [10828.300625] LustreError: 452299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353773 with bad export cookie 12168940348229417308 [10828.305230] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10828.305573] LustreError: 452299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10828.380477] LustreError: 454949:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10828.383990] LustreError: 454949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10828.443897] Lustre: server umount lustre-MDT0001 complete [10840.252410] LustreError: 455149:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10840.255578] LustreError: 455149:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10840.315982] Lustre: server umount lustre-OST0000 complete [10845.229250] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:29:50 (1763353790) [10845.695946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10848.742508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.012085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10850.848934] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10854.371780] LustreError: 455977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.377729] LustreError: 455977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10854.726660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10854.930310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10856.606883] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10859.007095] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10860.508519] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10863.351566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10863.393438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10863.476936] Lustre: Found index 0 for lustre-OST0000, updating log [10863.557403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10865.926059] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10868.326457] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10873.832457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10880.994021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10881.000172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10881.006457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10881.008992] Lustre: Skipped 2 previous similar messages [10886.703111] LustreError: 458258:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10886.706236] LustreError: 458258:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10886.758542] Lustre: server umount lustre-OST0000 complete [10887.901696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10890.721216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10890.728218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.732126] Lustre: Skipped 1 previous similar message [10890.735131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10890.737750] Lustre: Skipped 3 previous similar messages [10894.783850] LustreError: 458513:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10894.786593] LustreError: 458513:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10894.816448] Lustre: server umount lustre-MDT0000 complete [10896.490323] LustreError: 457594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353841 with bad export cookie 12168940348229418701 [10896.493548] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10896.495161] LustreError: 457594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10896.643525] Lustre: server umount lustre-MDT0001 complete [10900.334805] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:30:45 (1763353845) [10900.761348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10901.197756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10901.626963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10902.010934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10902.406287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10902.822825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10903.224538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10903.651361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10904.057304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10904.475172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10904.884570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10905.284757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10905.702583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10906.136425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10906.561802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10906.965956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10907.376432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10907.816622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10908.231427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10908.627486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10909.039258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10909.463329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10909.895067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10910.305459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10910.742530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10911.164233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10911.618226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10912.051125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10912.474122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10912.901845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10913.316389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10913.742203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10914.193683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10914.619572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10915.069383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10915.484475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10915.893166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10916.328979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10916.742871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10917.163570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10917.574690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10918.007283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10918.411986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10918.835252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10919.295824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10919.806105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10920.258995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10920.695134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10921.142425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10921.587644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10921.997575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10924.961101] LNet: 1 local NIs in recovery (showing 1): 192.168.201.156@tcp [10925.152058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.328447] LustreError: 462100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10925.359226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10926.885856] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10930.486097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10932.412279] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10934.794786] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10936.334621] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10939.102097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10939.150334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10939.259092] Lustre: Found index 0 for lustre-OST0000, updating log [10941.825438] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10944.217438] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10945.155127] LustreError: 464392:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10945.158792] LustreError: 464392:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10945.202130] Lustre: server umount lustre-OST0000 complete [10951.138351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10951.141751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10951.148887] Lustre: Skipped 1 previous similar message [10951.151452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10951.154034] Lustre: Skipped 2 previous similar messages [10959.844826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10959.847824] Lustre: Skipped 10 previous similar messages [10961.376164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10961.412375] LustreError: 464597:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10961.415129] LustreError: 464597:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10961.447914] Lustre: server umount lustre-MDT0000 complete [10961.889785] LustreError: 462118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10961.897683] LustreError: 462118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10963.037838] LustreError: 462081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353908 with bad export cookie 12168940348229419772 [10963.038517] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10963.044157] LustreError: 462081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10963.185228] Lustre: server umount lustre-MDT0001 complete [10964.413355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10971.303646] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [10975.839022] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10980.381390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10983.187074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10985.748885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10988.492066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10990.497309] LNet: 1 local NIs in recovery (showing 1): 192.168.201.156@tcp [10991.376358] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:32:16 (1763353936) [11000.943993] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11002.132080] Key type lgssc unregistered [11002.269700] LNet: 468359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11002.273573] LNetError: 468359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11126.184163] LNet: Removed LNI 192.168.201.156@tcp [11126.520927] Key type .llcrypt unregistered [11126.522212] Key type ._llcrypt unregistered [11134.217733] Key type ._llcrypt registered [11134.219343] Key type .llcrypt registered [11134.281428] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11134.796981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11134.812213] alg: No test for adler32 (adler32-zlib) [11135.716917] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [11135.830270] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11137.424166] Key type lgssc registered [11137.927451] Lustre: Echo OBD driver; http://www.lustre.org/ [11139.787662] LNet: Added LNI 192.168.201.156@tcp100 [8/256/0/180] [11140.179524] LNet: Added LNI 192.168.201.156@tcp101 [8/256/0/180] [11140.579967] LNet: Added LNI 192.168.201.156@tcp102 [8/256/0/180] [11140.945326] LNet: Added LNI 192.168.201.156@tcp103 [8/256/0/180] [11141.338576] LNet: Added LNI 192.168.201.156@tcp104 [8/256/0/180] [11141.720060] LNet: Added LNI 192.168.201.156@tcp105 [8/256/0/180] [11142.083445] LNet: Added LNI 192.168.201.156@tcp106 [8/256/0/180] [11142.431585] LNet: Added LNI 192.168.201.156@tcp107 [8/256/0/180] [11142.837279] LNet: Added LNI 192.168.201.156@tcp108 [8/256/0/180] [11143.211787] LNet: Added LNI 192.168.201.156@tcp109 [8/256/0/180] [11143.563183] LNet: Added LNI 192.168.201.156@tcp110 [8/256/0/180] [11143.895963] LNet: Added LNI 192.168.201.156@tcp111 [8/256/0/180] [11144.240991] LNet: Added LNI 192.168.201.156@tcp112 [8/256/0/180] [11144.600435] LNet: Added LNI 192.168.201.156@tcp113 [8/256/0/180] [11144.939155] LNet: Added LNI 192.168.201.156@tcp114 [8/256/0/180] [11145.303178] LNet: Added LNI 192.168.201.156@tcp115 [8/256/0/180] [11145.674574] LNet: Added LNI 192.168.201.156@tcp116 [8/256/0/180] [11146.053577] LNet: Added LNI 192.168.201.156@tcp117 [8/256/0/180] [11146.423728] LNet: Added LNI 192.168.201.156@tcp118 [8/256/0/180] [11146.779610] LNet: Added LNI 192.168.201.156@tcp119 [8/256/0/180] [11147.135996] LNet: Added LNI 192.168.201.156@tcp120 [8/256/0/180] [11147.521661] LNet: Added LNI 192.168.201.156@tcp121 [8/256/0/180] [11147.889627] LNet: Added LNI 192.168.201.156@tcp122 [8/256/0/180] [11148.264264] LNet: Added LNI 192.168.201.156@tcp123 [8/256/0/180] [11148.633970] LNet: Added LNI 192.168.201.156@tcp124 [8/256/0/180] [11148.992887] LNet: Added LNI 192.168.201.156@tcp125 [8/256/0/180] [11149.360480] LNet: Added LNI 192.168.201.156@tcp126 [8/256/0/180] [11149.726103] LNet: Added LNI 192.168.201.156@tcp127 [8/256/0/180] [11150.129443] LNet: Added LNI 192.168.201.156@tcp128 [8/256/0/180] [11150.560602] LNet: Added LNI 192.168.201.156@tcp129 [8/256/0/180] [11150.931277] LNet: Added LNI 192.168.201.156@tcp130 [8/256/0/180] [11151.279244] LNet: Added LNI 192.168.201.156@tcp131 [8/256/0/180] [11151.658594] LNet: Added LNI 192.168.201.156@tcp132 [8/256/0/180] [11152.025642] LNet: Added LNI 192.168.201.156@tcp133 [8/256/0/180] [11152.433203] LNet: Added LNI 192.168.201.156@tcp134 [8/256/0/180] [11152.831084] LNet: Added LNI 192.168.201.156@tcp135 [8/256/0/180] [11155.664928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11155.693604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11155.700959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11156.752317] Lustre: 471348:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.156@tcp: can't add NID 192.168.201.156@tcp130: rc = -75 [11156.781327] LNetError: 471369:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11156.786377] LNetError: Protocol error connecting to 192.168.201.156@tcp121 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11156.791411] LNetError: Skipped 2 previous similar messages [11156.842597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11156.857920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11156.899667] Lustre: lustre-MDT0000: new disk, initializing [11156.955153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11156.965758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11157.344409] LNet: 1 local NIs in recovery (showing 1): 192.168.201.156@tcp121 [11157.345830] LNetError: 471375:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11157.352222] LNetError: 471375:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [11157.356773] LNetError: Protocol error connecting to 192.168.201.156@tcp107 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11158.774986] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11159.394334] LNetError: 471375:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11159.399342] LNetError: 471375:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 5 previous similar messages [11159.403551] LNetError: Protocol error connecting to 192.168.201.156@tcp121 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11159.408591] LNetError: Skipped 5 previous similar messages [11163.490208] LNetError: 468809:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11163.495023] LNetError: Protocol error connecting to 192.168.201.156@tcp121 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11164.210453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11164.238648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11164.296388] Lustre: 471366:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11164.328828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11164.331360] Lustre: Skipped 1 previous similar message [11164.372152] Lustre: lustre-MDT0001: new disk, initializing [11164.391921] Lustre: 472152:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.156@tcp: can't add NID 192.168.201.156@tcp130: rc = -75 [11164.395901] Lustre: 472152:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11164.476841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11164.494382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11164.500162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11166.059904] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11169.660134] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11171.239712] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11171.681498] LNetError: 468811:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11171.685841] LNetError: Protocol error connecting to 192.168.201.156@tcp121 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11174.018082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11174.053950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11174.182921] Lustre: lustre-OST0000: new disk, initializing [11174.185874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11174.227479] Lustre: 473124:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.156@tcp: can't add NID 192.168.201.156@tcp130: rc = -75 [11174.231799] Lustre: 473124:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [11174.255865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11176.586395] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11180.310493] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11181.568315] Lustre: 473783:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.156@tcp: can't add NID 192.168.201.156@tcp130: rc = -75 [11181.572555] Lustre: 473783:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11181.603672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11181.610376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11181.673473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11186.017347] LNetError: 471371:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11186.021751] LNetError: Protocol error connecting to 192.168.201.156@tcp114 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11186.660829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11186.664510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11186.670503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11186.673109] Lustre: Skipped 1 previous similar message [11189.615157] LustreError: 473964:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11189.663841] Lustre: server umount lustre-OST0000 complete [11191.777819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11191.778656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11191.782132] Lustre: Skipped 2 previous similar messages [11196.897547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11196.901966] Lustre: Skipped 2 previous similar messages [11197.322061] LustreError: 474166:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11197.324958] LustreError: 474166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11197.374880] Lustre: server umount lustre-MDT0000 complete [11198.867319] LustreError: 472151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354144 with bad export cookie 12260094439524841730 [11198.869750] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11198.872355] LustreError: 472151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11198.944381] LustreError: 474367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11198.947285] LustreError: 474367:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11199.012225] Lustre: server umount lustre-MDT0001 complete [11203.295702] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11203.425638] LNetError: 468808:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11203.430813] LNetError: 468808:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 9 previous similar messages [11203.434805] LNetError: Protocol error connecting to 192.168.201.156@tcp107 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11203.439657] LNetError: Skipped 9 previous similar messages [11204.569878] Key type lgssc unregistered [11204.761832] LNet: 474906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11204.766791] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11204.770537] LNet: Removed LNI 192.168.201.156@tcp [11205.792193] LNet: Removed LNI 192.168.201.156@tcp100 [11205.794344] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11205.797919] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11205.801639] LNet: Removed LNI 192.168.201.156@tcp101 [11206.816173] LNet: Removed LNI 192.168.201.156@tcp102 [11206.818324] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11206.821479] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11206.825221] LNet: Removed LNI 192.168.201.156@tcp103 [11207.847493] LNet: Removed LNI 192.168.201.156@tcp104 [11208.864577] LNet: Removed LNI 192.168.201.156@tcp105 [11208.866377] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11208.871153] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11208.876069] LNet: Removed LNI 192.168.201.156@tcp106 [11209.888139] LNet: Removed LNI 192.168.201.156@tcp107 [11210.912408] LNet: Removed LNI 192.168.201.156@tcp108 [11211.936153] LNet: Removed LNI 192.168.201.156@tcp109 [11212.960212] LNet: Removed LNI 192.168.201.156@tcp110 [11212.962288] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11212.965554] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11212.969292] LNet: Removed LNI 192.168.201.156@tcp111 [11213.984157] LNet: Removed LNI 192.168.201.156@tcp112 [11215.008170] LNet: Removed LNI 192.168.201.156@tcp113 [11216.032194] LNet: Removed LNI 192.168.201.156@tcp114 [11217.056238] LNet: Removed LNI 192.168.201.156@tcp115 [11218.080211] LNet: Removed LNI 192.168.201.156@tcp116 [11219.104165] LNet: Removed LNI 192.168.201.156@tcp117 [11220.128155] LNet: Removed LNI 192.168.201.156@tcp118 [11221.152882] LNet: Removed LNI 192.168.201.156@tcp119 [11221.154979] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11221.158685] LNetError: 474906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11221.162479] LNet: Removed LNI 192.168.201.156@tcp120 [11221.164634] LNet: Removed LNI 192.168.201.156@tcp121 [11222.176193] LNet: Removed LNI 192.168.201.156@tcp122 [11222.179422] LNet: Removed LNI 192.168.201.156@tcp123 [11222.181235] LNet: Removed LNI 192.168.201.156@tcp124 [11222.183206] LNet: Removed LNI 192.168.201.156@tcp125 [11222.185274] LNet: Removed LNI 192.168.201.156@tcp126 [11222.187145] LNet: Removed LNI 192.168.201.156@tcp127 [11222.189234] LNet: Removed LNI 192.168.201.156@tcp128 [11222.191065] LNet: Removed LNI 192.168.201.156@tcp129 [11222.193075] LNet: Removed LNI 192.168.201.156@tcp130 [11222.195234] LNet: Removed LNI 192.168.201.156@tcp131 [11222.196989] LNet: Removed LNI 192.168.201.156@tcp132 [11222.199087] LNet: Removed LNI 192.168.201.156@tcp133 [11222.200877] LNet: Removed LNI 192.168.201.156@tcp134 [11222.209454] LNet: Removed LNI 192.168.201.156@tcp135 [11222.553636] Key type .llcrypt unregistered [11222.555187] Key type ._llcrypt unregistered [11231.952804] Key type ._llcrypt registered [11231.954341] Key type .llcrypt registered [11231.999667] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [11239.203411] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11239.570553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11239.618089] alg: No test for adler32 (adler32-zlib) [11240.503723] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [11240.604244] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11242.200243] Key type lgssc registered [11242.650193] Lustre: Echo OBD driver; http://www.lustre.org/ [11246.432906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11249.430054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11251.972729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11254.620180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11257.411301] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:36:42 (1763354202) [11264.469719] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11265.691516] Key type lgssc unregistered [11265.838375] LNet: 478920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11265.841688] LNetError: 478920:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11265.853427] LNet: Removed LNI 192.168.201.156@tcp [11266.197495] Key type .llcrypt unregistered [11266.198979] Key type ._llcrypt unregistered [11273.791160] Key type ._llcrypt registered [11273.792829] Key type .llcrypt registered [11273.843725] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11274.249091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11274.267485] alg: No test for adler32 (adler32-zlib) [11275.159563] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [11275.267623] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11276.864230] Key type lgssc registered [11277.321889] Lustre: Echo OBD driver; http://www.lustre.org/ [11281.353599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11281.374422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11281.382320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11282.483754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11282.507644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11282.552045] Lustre: lustre-MDT0000: new disk, initializing [11282.596541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11282.609899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11284.265408] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11289.394699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11289.435378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11289.493481] Lustre: 480092:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11289.524776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11289.527686] Lustre: Skipped 1 previous similar message [11289.567515] Lustre: lustre-MDT0001: new disk, initializing [11289.635469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11289.654561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11289.662978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11291.298632] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11295.070353] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11296.652250] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11299.572448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11299.605157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11299.733891] Lustre: lustre-OST0000: new disk, initializing [11299.736695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11299.782264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11302.262843] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11306.138840] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11307.018564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11307.022476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11307.074747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11307.620458] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing lnet_if_list [11309.662407] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11309.680821] LNet: Added LNI 192.168.201.156@tcp10 [8/256/0/180] [11311.808908] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11311.822868] LNet: Added LNI 192.168.201.156@tcp11 [8/256/0/180] [11313.339441] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11313.355273] LNet: Added LNI 192.168.201.156@tcp12 [8/256/0/180] [11314.945743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11314.960039] LNet: Added LNI 192.168.201.156@tcp13 [8/256/0/180] [11316.628200] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11316.642306] LNet: Added LNI 192.168.201.156@tcp14 [8/256/0/180] [11318.319506] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11318.332605] LNet: Added LNI 192.168.201.156@tcp15 [8/256/0/180] [11320.146505] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11320.160714] LNet: Added LNI 192.168.201.156@tcp16 [8/256/0/180] [11322.038616] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11322.053957] LNet: Added LNI 192.168.201.156@tcp17 [8/256/0/180] [11323.931238] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11323.946439] LNet: Added LNI 192.168.201.156@tcp18 [8/256/0/180] [11325.760341] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11325.773800] LNet: Added LNI 192.168.201.156@tcp19 [8/256/0/180] [11327.720873] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11327.736718] LNet: Added LNI 192.168.201.156@tcp20 [8/256/0/180] [11329.716895] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11329.733051] LNet: Added LNI 192.168.201.156@tcp21 [8/256/0/180] [11331.774808] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11331.793026] LNet: Added LNI 192.168.201.156@tcp22 [8/256/0/180] [11333.998650] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11334.013335] LNet: Added LNI 192.168.201.156@tcp23 [8/256/0/180] [11336.149414] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11336.163614] LNet: Added LNI 192.168.201.156@tcp24 [8/256/0/180] [11338.345685] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11338.359631] LNet: Added LNI 192.168.201.156@tcp25 [8/256/0/180] [11340.607256] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11340.621474] LNet: Added LNI 192.168.201.156@tcp26 [8/256/0/180] [11342.996525] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11343.013398] LNet: Added LNI 192.168.201.156@tcp27 [8/256/0/180] [11343.021954] LNetError: 479354:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11343.030424] LNetError: Protocol error connecting to 192.168.201.156@tcp26 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11345.350077] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11345.363980] LNet: Added LNI 192.168.201.156@tcp28 [8/256/0/180] [11345.367902] LNetError: 479351:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11345.372754] LNetError: Protocol error connecting to 192.168.201.156@tcp26 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11347.749304] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11347.765094] LNet: Added LNI 192.168.201.156@tcp29 [8/256/0/180] [11347.770040] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11347.777734] LNetError: Protocol error connecting to 192.168.201.156@tcp28 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11348.000561] LNet: 1 local NIs in recovery (showing 1): 192.168.201.156@tcp28 [11350.177930] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11350.192492] LNet: Added LNI 192.168.201.156@tcp30 [8/256/0/180] [11351.075215] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11351.079610] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [11351.083045] LNetError: Protocol error connecting to 192.168.201.156@tcp28 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11351.087413] LNetError: Skipped 3 previous similar messages [11352.589616] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11352.606254] LNet: Added LNI 192.168.201.156@tcp31 [8/256/0/180] [11355.111770] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11355.126643] LNet: Added LNI 192.168.201.156@tcp32 [8/256/0/180] [11355.172171] LNetError: 479354:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11355.176525] LNetError: 479354:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [11355.180487] LNetError: Protocol error connecting to 192.168.201.156@tcp30 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11355.185088] LNetError: Skipped 2 previous similar messages [11357.689598] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11357.706860] LNet: Added LNI 192.168.201.156@tcp33 [8/256/0/180] [11360.533535] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11360.548156] LNet: Added LNI 192.168.201.156@tcp34 [8/256/0/180] [11363.152788] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11363.170409] LNet: Added LNI 192.168.201.156@tcp35 [8/256/0/180] [11363.362188] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11363.368200] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 6 previous similar messages [11363.371783] LNetError: Protocol error connecting to 192.168.201.156@tcp32 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11363.376378] LNetError: Skipped 6 previous similar messages [11365.821420] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11365.835251] LNet: Added LNI 192.168.201.156@tcp36 [8/256/0/180] [11368.659404] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11368.676179] LNet: Added LNI 192.168.201.156@tcp37 [8/256/0/180] [11371.682536] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11371.696090] LNet: Added LNI 192.168.201.156@tcp38 [8/256/0/180] [11374.951116] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11374.968509] LNet: Added LNI 192.168.201.156@tcp39 [8/256/0/180] [11378.311894] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11378.326877] LNet: Added LNI 192.168.201.156@tcp40 [8/256/0/180] [11379.746471] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11379.753986] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 25 previous similar messages [11379.757893] LNetError: Protocol error connecting to 192.168.201.156@tcp39 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11379.765503] LNetError: Skipped 25 previous similar messages [11381.439656] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11381.452885] LNet: Added LNI 192.168.201.156@tcp41 [8/256/0/180] [11384.487929] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11384.502359] LNet: Added LNI 192.168.201.156@tcp42 [8/256/0/180] [11387.597487] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11387.611960] LNet: Added LNI 192.168.201.156@tcp43 [8/256/0/180] [11391.056049] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11391.074648] LNet: Added LNI 192.168.201.156@tcp44 [8/256/0/180] [11394.451634] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11394.465569] LNet: Added LNI 192.168.201.156@tcp45 [8/256/0/180] [11398.067608] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11398.085369] LNet: Added LNI 192.168.201.156@tcp46 [8/256/0/180] [11401.689947] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11401.713365] LNet: Added LNI 192.168.201.156@tcp47 [8/256/0/180] [11405.349164] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11405.363778] LNet: Added LNI 192.168.201.156@tcp48 [8/256/0/180] [11408.418091] LNet: 9 local NIs in recovery (showing 9): 192.168.201.156@tcp46, 192.168.201.156@tcp42, 192.168.201.156@tcp37, 192.168.201.156@tcp43, 192.168.201.156@tcp44, 192.168.201.156@tcp33, 192.168.201.156@tcp39, 192.168.201.156@tcp32, 192.168.201.156@tcp30 [11408.858476] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11408.874211] LNet: Added LNI 192.168.201.156@tcp49 [8/256/0/180] [11412.513392] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.156 [11412.522256] LNetError: 479352:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 45 previous similar messages [11412.528803] LNetError: Protocol error connecting to 192.168.201.156@tcp35 at host 192.168.201.156:988: is it running a compatible version of Lustre? [11412.534765] LNetError: Skipped 45 previous similar messages [11412.550640] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11412.569694] LNet: Added LNI 192.168.201.156@tcp50 [8/256/0/180] [11424.740031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11424.740330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11424.745983] LustreError: Skipped 1 previous similar message [11424.753936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11424.758399] Lustre: Skipped 1 previous similar message [11427.956159] LustreError: 504293:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11428.013959] Lustre: server umount lustre-OST0000 complete [11429.858342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11429.863633] Lustre: Skipped 1 previous similar message [11429.866962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11429.869234] Lustre: Skipped 1 previous similar message [11434.978231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11434.981380] Lustre: Skipped 1 previous similar message [11435.871925] LustreError: 504494:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11435.874405] LustreError: 504494:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11435.926500] Lustre: server umount lustre-MDT0000 complete [11437.629182] LustreError: 480871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354382 with bad export cookie 16026977824207083574 [11437.631163] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11437.634228] LustreError: 480871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11437.709572] LustreError: 504695:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11437.711669] LustreError: 504695:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11437.780488] Lustre: server umount lustre-MDT0001 complete [11441.965735] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11443.245261] Key type lgssc unregistered [11443.448919] LNet: 505234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11443.457564] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11443.462723] LNet: Removed LNI 192.168.201.156@tcp [11444.512290] LNet: Removed LNI 192.168.201.156@tcp10 [11444.515288] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11444.521829] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11444.526161] LNet: Removed LNI 192.168.201.156@tcp11 [11445.536201] LNet: Removed LNI 192.168.201.156@tcp12 [11445.538173] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11445.544015] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11445.548513] LNet: Removed LNI 192.168.201.156@tcp13 [11446.560268] LNet: Removed LNI 192.168.201.156@tcp14 [11447.584199] LNet: Removed LNI 192.168.201.156@tcp15 [11447.586637] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11447.589862] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11447.593557] LNet: Removed LNI 192.168.201.156@tcp16 [11448.608842] LNet: Removed LNI 192.168.201.156@tcp17 [11449.632197] LNet: Removed LNI 192.168.201.156@tcp18 [11450.656232] LNet: Removed LNI 192.168.201.156@tcp19 [11451.680247] LNet: Removed LNI 192.168.201.156@tcp20 [11451.681997] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11451.684806] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11451.687940] LNet: Removed LNI 192.168.201.156@tcp21 [11452.704126] LNet: Removed LNI 192.168.201.156@tcp22 [11453.728196] LNet: Removed LNI 192.168.201.156@tcp23 [11454.752153] LNet: Removed LNI 192.168.201.156@tcp24 [11455.776205] LNet: Removed LNI 192.168.201.156@tcp25 [11456.800254] LNet: Removed LNI 192.168.201.156@tcp26 [11457.824220] LNet: Removed LNI 192.168.201.156@tcp27 [11458.849803] LNet: Removed LNI 192.168.201.156@tcp28 [11459.872213] LNet: Removed LNI 192.168.201.156@tcp29 [11459.873995] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11459.877573] LNetError: 505234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11459.881515] LNet: Removed LNI 192.168.201.156@tcp30 [11460.896251] LNet: Removed LNI 192.168.201.156@tcp31 [11460.897985] LNet: Removed LNI 192.168.201.156@tcp32 [11460.899731] LNet: Removed LNI 192.168.201.156@tcp33 [11461.920165] LNet: Removed LNI 192.168.201.156@tcp34 [11462.944216] LNet: Removed LNI 192.168.201.156@tcp35 [11463.968177] LNet: Removed LNI 192.168.201.156@tcp36 [11463.970045] LNet: Removed LNI 192.168.201.156@tcp37 [11464.992210] LNet: Removed LNI 192.168.201.156@tcp38 [11466.016202] LNet: Removed LNI 192.168.201.156@tcp39 [11467.040201] LNet: Removed LNI 192.168.201.156@tcp40 [11468.064112] LNet: Removed LNI 192.168.201.156@tcp41 [11468.065384] LNet: Removed LNI 192.168.201.156@tcp42 [11468.067106] LNet: Removed LNI 192.168.201.156@tcp43 [11468.068252] LNet: Removed LNI 192.168.201.156@tcp44 [11469.088187] LNet: Removed LNI 192.168.201.156@tcp45 [11470.112653] LNet: Removed LNI 192.168.201.156@tcp46 [11471.136254] LNet: Removed LNI 192.168.201.156@tcp47 [11472.160297] LNet: Removed LNI 192.168.201.156@tcp48 [11473.184550] LNet: Removed LNI 192.168.201.156@tcp49 [11473.192892] LNet: Removed LNI 192.168.201.156@tcp50 [11473.514643] LNetError: 505298:0:(module.c:692:libcfs_exit()) Portals memory leaked: 376 bytes [11473.537573] Key type .llcrypt unregistered [11473.539198] Key type ._llcrypt unregistered [11482.244412] Key type ._llcrypt registered [11482.246029] Key type .llcrypt registered [11482.295718] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11482.760828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11482.768368] alg: No test for adler32 (adler32-zlib) [11483.650119] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [11483.753547] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11485.344148] Key type lgssc registered [11485.799016] Lustre: Echo OBD driver; http://www.lustre.org/ [11489.065439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11489.074230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.309763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11491.860573] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11495.339812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11495.479807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11496.947124] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11499.040873] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11500.502911] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11503.312217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11503.471675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11505.909494] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11508.130757] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11524.318824] loop0: detected capacity change from 0 to 204800000 [11524.380623] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11524.385891] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11524.392730] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11524.418345] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11526.965408] loop0: detected capacity change from 0 to 204800000 [11527.028665] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11527.033606] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11527.040788] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11527.107095] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11529.204092] loop0: detected capacity change from 0 to 204800000 [11531.340308] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11531.358365] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11531.409198] Lustre: fs15246-OST0000: new disk, initializing [11531.412161] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11531.434171] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11533.804868] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11538.087787] loop1: detected capacity change from 0 to 204800000 [11540.308426] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11540.332942] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11540.373990] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11540.401463] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11540.440468] Lustre: fs15246-MDT0000: new disk, initializing [11540.495173] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11540.507641] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11540.555430] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11541.601338] Lustre: 510854:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000aa002918 x1849011133776896/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763354502 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11542.056195] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11546.658553] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11546.662422] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11546.670862] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11550.692038] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.697658] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11555.810843] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11556.733799] LustreError: 511978:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [11556.755588] Lustre: server umount fs15246-MDT0000 complete [11560.623327] LustreError: 512480:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [11560.625620] LustreError: 512480:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11560.639824] Lustre: server umount fs15246-OST0000 complete [11566.054028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11566.056236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11566.059672] Lustre: Skipped 1 previous similar message [11566.064175] Lustre: Skipped 1 previous similar message [11570.674709] LustreError: 512982:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11570.677470] LustreError: 512982:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [11570.726304] Lustre: server umount lustre-OST0000 complete [11572.193960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11572.196814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11572.203048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11576.293693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11576.294034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.311775] LustreError: 513184:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11578.314248] LustreError: 513184:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11578.355730] Lustre: server umount lustre-MDT0000 complete [11579.861449] LustreError: 506441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354525 with bad export cookie 9103875769006995714 [11579.866084] LustreError: 506441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11579.866935] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11580.007566] Lustre: server umount lustre-MDT0001 complete [11583.986511] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11585.133593] Key type lgssc unregistered [11585.278347] LNet: 513917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11585.281180] LNetError: 513917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11585.289434] LNet: Removed LNI 192.168.201.156@tcp [11585.623364] Key type .llcrypt unregistered [11585.624964] Key type ._llcrypt unregistered [11594.030651] Key type ._llcrypt registered [11594.032356] Key type .llcrypt registered [11594.084552] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11594.518211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11594.580387] alg: No test for adler32 (adler32-zlib) [11595.458253] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [11595.556084] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11597.152213] Key type lgssc registered [11597.620135] Lustre: Echo OBD driver; http://www.lustre.org/ [11600.638961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11602.732358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11604.931655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11606.949766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11613.374478] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [11617.575024] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11621.682052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11624.525817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11627.506075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11630.452670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11633.491717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11633.528670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11633.535355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11634.658572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11634.679637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11634.726798] Lustre: lustre-MDT0000: new disk, initializing [11634.781137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11634.790153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11636.369059] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11641.641183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11641.675376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11641.714199] Lustre: 518320:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11641.740838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11641.743713] Lustre: Skipped 1 previous similar message [11641.784753] Lustre: lustre-MDT0001: new disk, initializing [11641.829343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11641.852886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11641.857275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11643.447156] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11647.178577] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11648.873388] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11652.253810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11652.306280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11652.430991] Lustre: lustre-OST0000: new disk, initializing [11652.434490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11652.475408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11655.088194] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11658.762294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11658.766890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11658.831393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11659.253667] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11661.237741] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11661.341090] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11663.295526] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11663.401595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11668.962826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11668.964762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11668.966596] LustreError: Skipped 1 previous similar message [11668.973149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11668.977472] Lustre: Skipped 1 previous similar message [11670.451146] LustreError: 521255:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11670.497279] Lustre: server umount lustre-OST0000 complete [11672.544844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11672.547601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11672.552707] Lustre: Skipped 1 previous similar message [11672.555095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11674.081837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11674.082287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11674.090208] Lustre: Skipped 1 previous similar message [11679.202758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11679.206377] Lustre: Skipped 1 previous similar message [11684.321570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11684.324680] Lustre: Skipped 1 previous similar message [11686.368134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11686.412199] LustreError: 521456:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11686.415866] LustreError: 521456:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11686.444205] Lustre: server umount lustre-MDT0000 complete [11687.949429] LustreError: 519101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354633 with bad export cookie 11397005148873571022 [11687.953228] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11687.955125] LustreError: 519101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11688.025718] LustreError: 521660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11688.028373] LustreError: 521660:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11688.085978] Lustre: server umount lustre-MDT0001 complete [11698.416237] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [11703.416194] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11707.574667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11710.552530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11713.232195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11715.950324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11716.748591] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11510 sec ========= 23:44:21 (1763354661) [11717.535556] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:44:22 (1763354662) === [11718.931968] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:44:23 (1763354663) === [11734.781865] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11735.996663] Key type lgssc unregistered [11736.150545] LNet: 526510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11736.154883] LNetError: 526510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11736.162627] LNet: Removed LNI 192.168.201.156@tcp [11736.511575] Key type .llcrypt unregistered [11736.513341] Key type ._llcrypt unregistered