[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1309742654 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003241] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.008491] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010018] pid_max: default: 32768 minimum: 301 [ 0.012092] LSM: Security Framework initializing [ 0.013051] Yama: becoming mindful. [ 0.013972] SELinux: Initializing. [ 0.015058] *** VALIDATE selinux *** [ 0.022665] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027199] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030127] *** VALIDATE tmpfs *** [ 0.032120] *** VALIDATE proc *** [ 0.033265] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037037] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.043000] debug: unmapping init [mem 0xffffffffb0659000-0xffffffffb0660fff] [ 0.045250] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048009] ... bit width: 48 [ 0.048963] ... generic registers: 4 [ 0.049018] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053339] rcu: Hierarchical SRCU implementation. [ 0.055607] smp: Bringing up secondary CPUs ... [ 0.056579] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.062449] smp: Brought up 1 node, 4 CPUs [ 0.064025] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.121784] node 0 deferred pages initialised in 54ms [ 0.126038] devtmpfs: initialized [ 0.127388] x86/mm: Memory block size: 128MB [ 0.129951] gcov: version magic: 0x41383552 [ 0.133387] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135186] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138462] pinctrl core: initialized pinctrl subsystem [ 0.140172] [ 0.140626] ************************************************************* [ 0.143014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144012] ** ** [ 0.147018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149018] ** ** [ 0.152019] ** This means that this kernel is built to expose internal ** [ 0.155014] ** IOMMU data structures, which may compromise security on ** [ 0.157013] ** your system. ** [ 0.160026] ** ** [ 0.162014] ** If you see this message and you are not debugging the ** [ 0.165017] ** kernel, report this immediately to your vendor! ** [ 0.167015] ** ** [ 0.169016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172014] ************************************************************* [ 0.174969] NET: Registered protocol family 16 [ 0.177659] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183165] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187187] cpuidle: using governor menu [ 0.189000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.192071] PCI: Using configuration type 1 for base access [ 0.203146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.229076] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232073] cryptd: max_cpu_qlen set to 1000 [ 0.233267] ACPI: Added _OSI(Module Device) [ 0.234015] ACPI: Added _OSI(Processor Device) [ 0.235013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242959] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250530] ACPI: Interpreter enabled [ 0.252069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.254014] ACPI: Using IOAPIC for interrupt routing [ 0.256108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260467] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273812] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287872] acpiphp: Slot [2] registered [ 0.290156] acpiphp: Slot [5] registered [ 0.291134] acpiphp: Slot [6] registered [ 0.293141] acpiphp: Slot [7] registered [ 0.294324] acpiphp: Slot [8] registered [ 0.296141] acpiphp: Slot [9] registered [ 0.298144] acpiphp: Slot [10] registered [ 0.299234] acpiphp: Slot [3] registered [ 0.301171] acpiphp: Slot [4] registered [ 0.303107] acpiphp: Slot [11] registered [ 0.305115] acpiphp: Slot [12] registered [ 0.306138] acpiphp: Slot [13] registered [ 0.308118] acpiphp: Slot [14] registered [ 0.310158] acpiphp: Slot [15] registered [ 0.312118] acpiphp: Slot [16] registered [ 0.318175] acpiphp: Slot [17] registered [ 0.326147] acpiphp: Slot [18] registered [ 0.332141] acpiphp: Slot [19] registered [ 0.335153] acpiphp: Slot [20] registered [ 0.339185] acpiphp: Slot [21] registered [ 0.340118] acpiphp: Slot [22] registered [ 0.342107] acpiphp: Slot [23] registered [ 0.346141] acpiphp: Slot [24] registered [ 0.348138] acpiphp: Slot [25] registered [ 0.350165] acpiphp: Slot [26] registered [ 0.354123] acpiphp: Slot [27] registered [ 0.357139] acpiphp: Slot [28] registered [ 0.358108] acpiphp: Slot [29] registered [ 0.360110] acpiphp: Slot [30] registered [ 0.361083] acpiphp: Slot [31] registered [ 0.363058] PCI host bridge to bus 0000:00 [ 0.365026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.369026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.373024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.377026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.381025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.385041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.388182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.394523] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.399315] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.413016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.417019] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.419018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.421018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.424019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.428089] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.430813] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.433045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.434835] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.437000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.437000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.441000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.448903] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.453019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.493021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.527968] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.554030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.659017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.672021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.681950] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.688017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.693015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.708018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.716966] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.729016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.737018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.756017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.764853] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.779018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.788019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.812019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.830339] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.840017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.850017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.876018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.889804] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.892592] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.895380] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.898379] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.900307] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.905120] iommu: Default domain type: Passthrough [ 0.907159] SCSI subsystem initialized [ 0.909157] ACPI: bus type USB registered [ 0.910119] usbcore: registered new interface driver usbfs [ 0.912074] usbcore: registered new interface driver hub [ 0.914112] usbcore: registered new device driver usb [ 0.916530] pps_core: LinuxPPS API ver. 1 registered [ 0.918014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.921066] PTP clock support registered [ 0.923113] EDAC MC: Ver: 3.0.0 [ 0.924481] PCI: Using ACPI for IRQ routing [ 0.926124] NetLabel: Initializing [ 0.928011] NetLabel: domain hash size = 128 [ 0.929010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.931165] NetLabel: unlabeled traffic allowed by default [ 0.933299] vgaarb: loaded [ 0.935328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.937016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.943684] clocksource: Switched to clocksource kvm-clock [ 1.100370] VFS: Disk quotas dquot_6.6.0 [ 1.102362] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.105547] *** VALIDATE ramfs *** [ 1.107214] *** VALIDATE hugetlbfs *** [ 1.109445] pnp: PnP ACPI init [ 1.113135] pnp: PnP ACPI: found 6 devices [ 1.133710] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.137469] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.139590] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.142170] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.144705] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.147032] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.150199] NET: Registered protocol family 2 [ 1.152442] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.157458] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.161069] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.166946] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.170843] TCP: Hash tables configured (established 65536 bind 65536) [ 1.173955] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.177197] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.179941] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.182775] NET: Registered protocol family 1 [ 1.185214] RPC: Registered named UNIX socket transport module. [ 1.187401] RPC: Registered udp transport module. [ 1.188589] RPC: Registered tcp transport module. [ 1.190131] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.191915] NET: Registered protocol family 44 [ 1.193364] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.194952] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.196612] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.198564] PCI: CLS 0 bytes, default 64 [ 1.200215] Unpacking initramfs... [ 3.751602] debug: unmapping init [mem 0xffff9a38fcc54000-0xffff9a38fffbffff] [ 3.756132] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.758500] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.761388] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.374699] Initialise system trusted keyrings [ 4.376338] Key type blacklist registered [ 4.378598] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.391337] zbud: loaded [ 4.395444] *** VALIDATE nfs *** [ 4.396567] *** VALIDATE nfs4 *** [ 4.399068] pstore: using deflate compression [ 4.404259] Platform Keyring initialized [ 4.575132] NET: Registered protocol family 38 [ 4.577387] Key type asymmetric registered [ 4.579303] Asymmetric key parser 'x509' registered [ 4.581650] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.584946] io scheduler mq-deadline registered [ 4.586937] io scheduler kyber registered [ 4.588795] io scheduler bfq registered [ 4.590784] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.594151] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.598142] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.601506] ACPI: Power Button [PWRF] [ 4.607883] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.618258] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.656824] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.670314] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.695982] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.724933] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.754161] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.759317] Non-volatile memory driver v1.3 [ 4.761161] Linux agpgart interface v0.103 [ 4.807157] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 4.809755] vda: detected capacity change from 0 to 69627904 [ 4.832253] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.836143] vdb: detected capacity change from 0 to 1073741824 [ 4.858653] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.861103] vdc: detected capacity change from 0 to 2621440000 [ 4.886390] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.889688] vdd: detected capacity change from 0 to 2621440000 [ 4.910658] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.913768] vde: detected capacity change from 0 to 4294967296 [ 4.945436] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.948727] vdf: detected capacity change from 0 to 4294967296 [ 4.972464] libphy: Fixed MDIO Bus: probed [ 4.989047] usbcore: registered new interface driver usbserial_generic [ 4.991430] usbserial: USB Serial support registered for generic [ 4.994027] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.999398] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.001458] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.007421] mousedev: PS/2 mouse device common for all mice [ 5.011362] rtc_cmos 00:05: RTC can wake from S4 [ 5.014520] rtc_cmos 00:05: registered as rtc0 [ 5.016502] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.019337] intel_pstate: CPU model not supported [ 5.022140] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.024913] hid: raw HID events driver (C) Jiri Kosina [ 5.029167] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.031846] usbcore: registered new interface driver usbhid [ 5.039783] usbhid: USB HID core driver [ 5.040828] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.044601] drop_monitor: Initializing network drop monitor service [ 5.044749] Initializing XFRM netlink socket [ 5.045120] NET: Registered protocol family 10 [ 5.046440] Segment Routing with IPv6 [ 5.046481] NET: Registered protocol family 17 [ 5.051059] mpls_gso: MPLS GSO support [ 5.069155] RAS: Correctable Errors collector initialized. [ 5.071251] AVX version of gcm_enc/dec engaged. [ 5.073115] AES CTR mode by8 optimization enabled [ 5.195143] sched_clock: Marking stable (5195092404, 0)->(6592413181, -1397320777) [ 5.201352] registered taskstats version 1 [ 5.207409] Loading compiled-in X.509 certificates [ 5.213351] zswap: loaded using pool lzo/zbud [ 5.243852] Key type big_key registered [ 5.277813] Key type encrypted registered [ 5.279768] ima: No TPM chip found, activating TPM-bypass! [ 5.283388] ima: Allocated hash algorithm: sha1 [ 5.285298] ima: No architecture policies found [ 5.287395] evm: Initialising EVM extended attributes: [ 5.289760] evm: security.selinux [ 5.291259] evm: security.ima [ 5.292427] evm: security.capability [ 5.293985] evm: HMAC attrs: 0x1 [ 5.296774] rtc_cmos 00:05: setting system clock to 2026-06-02 06:07:28 UTC (1780380448) [ 5.305323] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 5.308885] debug: unmapping init [mem 0xffffffffb0382000-0xffffffffb0658fff] [ 5.318131] Write protecting the kernel read-only data: 28672k [ 5.328483] debug: unmapping init [mem 0xffffffffaea03000-0xffffffffaebfffff] [ 5.331930] debug: unmapping init [mem 0xffffffffaf314000-0xffffffffaf3fffff] [ 5.379752] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.390517] systemd[1]: Detected virtualization kvm. [ 5.396260] systemd[1]: Detected architecture x86-64. [ 5.398743] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.431174] systemd[1]: No hostname configured. [ 5.433615] systemd[1]: Set hostname to . [ 5.437194] random: systemd: uninitialized urandom read (16 bytes read) [ 5.441142] systemd[1]: Initializing machine ID from random generator. [ 5.588326] random: ln: uninitialized urandom read (6 bytes read) [ 5.793848] random: systemd: uninitialized urandom read (16 bytes read) [ 5.798676] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.805982] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.815617] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.044933] device-mapper: uevent: version 1.0.3 [ 8.047194] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.823750] virtio_net virtio0 ens2: renamed from eth0 [ 9.874942] random: fast init done [ 11.782482] hrtimer: interrupt took 8468531 ns [ 15.053593] scsi host0: ata_piix [ 16.089898] scsi host1: ata_piix [ 16.096538] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 16.102962] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.111069] random: crng init done [ 16.112994] random: 7 urandom warning(s) missed due to ratelimiting [ 19.297465] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.825873] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.119994] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.441909] SELinux: Disabled at runtime. [ 28.578391] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.593448] systemd[1]: Detected virtualization kvm. [ 28.600794] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.790388] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.802329] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.816064] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.821269] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.825796] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.837893] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.849320] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ 30.907215] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 33.236339] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.331783] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.407053] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.199616] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.312785] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 42.840544] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (12s / no limit)[ 43.847652] NFS: Registering the id_resolver key type [ 43.854662] Key type id_resolver registered [ 43.860605] Key type id_legacy registered [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg149-server login: [ 95.367492] libcfs: loading out-of-tree module taints kernel. [ 95.407953] Key type ._llcrypt registered [ 95.409175] Key type .llcrypt registered [ 95.496927] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 112.437314] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 113.176904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.194073] alg: No test for adler32 (adler32-zlib) [ 114.331504] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 114.763097] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 116.439370] Key type lgssc registered [ 117.451324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.402503] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.975444] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 184.150373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.195838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.538402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.628658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.943882] Lustre: lustre-MDT0000: new disk, initializing [ 187.213327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.304665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.813784] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.039526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.159518] Lustre: 6505:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.198477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 213.208815] Lustre: Skipped 1 previous similar message [ 213.318818] Lustre: lustre-MDT0001: new disk, initializing [ 213.382200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.406702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.421191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.068732] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.064374] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 232.688040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.943474] Lustre: lustre-OST0000: new disk, initializing [ 232.947756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.953905] Lustre: 8412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.006871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.162985] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.187064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.200721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.323567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.939244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.203087] Lustre: lustre-OST0001: new disk, initializing [ 255.206280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 255.220583] Lustre: 9466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.291974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.675497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.693522] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.782070] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.241620] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.338223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.769716] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.883457] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing check_logdir /tmp/testlogs/ [ 294.291346] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing yml_node [ 299.079320] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 301.695726] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 304.272185] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 306.199319] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 2 02:12:26 EDT 2026 [ 329.912606] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 331.850314] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.713947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 351.728789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.754748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.256075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.256836] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.264589] Lustre: Skipped 2 previous similar messages [ 353.283021] Lustre: Skipped 2 previous similar messages [ 355.601870] Lustre: server umount lustre-MDT0000 complete [ 359.498704] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780380802 with bad export cookie 6781899450222306877 [ 359.510918] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.488658] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.494312] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.501715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.501724] Lustre: Skipped 1 previous similar message [ 363.510062] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 363.546575] Lustre: Skipped 1 previous similar message [ 364.512736] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.528785] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 366.268747] Lustre: server umount lustre-MDT0001 complete [ 370.892513] Lustre: server umount lustre-OST0000 complete [ 375.926500] Lustre: server umount lustre-OST0001 complete [ 383.956954] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 393.368759] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 436.595129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.833481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 436.856821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 436.927660] Lustre: lustre-MDT0000: new disk, initializing [ 437.002874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.019891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 441.570239] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 453.462562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.584855] Lustre: 14979:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 453.647842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 453.657505] Lustre: Skipped 1 previous similar message [ 453.788376] Lustre: lustre-MDT0001: new disk, initializing [ 453.973950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.037180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 454.064068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.494829] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 469.068409] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 473.631711] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 481.770600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.108728] Lustre: lustre-OST0000: new disk, initializing [ 482.114127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 482.122690] Lustre: 16725:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 482.258031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.207646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 483.224157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 483.353502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 489.834563] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 498.356091] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 503.429502] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 503.658325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 508.237860] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 508.570976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 514.015981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 514.022273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.029992] LustreError: Skipped 1 previous similar message [ 514.031059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 514.031065] Lustre: Skipped 3 previous similar messages [ 514.059375] Lustre: Skipped 1 previous similar message [ 516.845501] Lustre: server umount lustre-OST0000 complete [ 524.262464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.291081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.293784] Lustre: Skipped 1 previous similar message [ 527.938645] Lustre: server umount lustre-MDT0000 complete [ 529.384098] LustreError: 14991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.406775] LustreError: 14991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 532.378752] LustreError: 14971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780380975 with bad export cookie 6781899450222308410 [ 532.388848] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.395950] LustreError: 14971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 532.855711] Lustre: server umount lustre-MDT0001 complete [ 537.698336] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:16:18 (1780380978) [ 540.753157] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 542.890527] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:16:23 (1780380983) [ 550.427534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.014530] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.153228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.514019] LustreError: 18990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.589774] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 566.743245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.756932] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.776032] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 572.552963] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 579.282763] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 584.433968] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 592.605113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 592.813136] LustreError: 20585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.070681] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 606.744336] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 616.933631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.944668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 616.966232] Lustre: Skipped 1 previous similar message [ 616.967465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 621.568828] Lustre: server umount lustre-OST0000 complete [ 632.289616] LustreError: 18984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.311150] LustreError: 18984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 633.680816] Lustre: server umount lustre-MDT0000 complete [ 638.120813] LustreError: 18969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381081 with bad export cookie 6781899450222309292 [ 638.132582] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.135261] LustreError: 18969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 638.713819] Lustre: server umount lustre-MDT0001 complete [ 651.356202] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 655.669333] Key type lgssc unregistered [ 656.041408] LNet: 22168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.057855] LNetError: 22168:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.080826] LNet: Removed LNI 192.168.201.149@tcp [ 657.475344] Key type .llcrypt unregistered [ 657.487782] Key type ._llcrypt unregistered [ 682.835795] Key type ._llcrypt registered [ 682.838088] Key type .llcrypt registered [ 682.905856] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 684.201063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 684.223990] alg: No test for adler32 (adler32-zlib) [ 685.841316] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 686.015909] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 687.975288] Key type lgssc registered [ 689.947371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 699.678662] loop: module loaded [ 699.816708] loop0: detected capacity change from 0 to 204800000 [ 700.145634] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.164973] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 700.559346] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.779212] loop0: detected capacity change from 0 to 204800000 [ 709.076442] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.094142] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.112972] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.886495] loop0: detected capacity change from 0 to 204800000 [ 723.021822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 723.054591] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.417204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 724.481289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 724.595275] Lustre: lustre-MDT0000: new disk, initializing [ 724.812841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 724.864916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 730.226283] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 741.282218] loop1: detected capacity change from 0 to 204800000 [ 748.064053] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.540374] Lustre: lustre-OST07c6: new disk, initializing [ 748.544304] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 748.549538] Lustre: Skipped 1 previous similar message [ 748.555337] Lustre: 25325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 748.675326] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 756.210552] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 756.405734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 756.425159] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 756.458636] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 762.551479] Lustre: 24330:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 762.561189] Lustre: 24330:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 762.569447] Lustre: 24330:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 762.575694] Lustre: 24330:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 762.579937] Lustre: 24330:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 762.584824] Lustre: 24330:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 781.635831] Lustre: 24329:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 781.643155] Lustre: 24329:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 781.652034] Lustre: 24329:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.657925] Lustre: 24329:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 781.661504] Lustre: 24329:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 781.665028] Lustre: 24329:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 787.167422] Lustre: 24330:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 787.186099] Lustre: 24330:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 787.193719] Lustre: 24330:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 787.202448] Lustre: 24330:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.212316] Lustre: 24330:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 787.220295] Lustre: 24330:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.233574] Lustre: 24330:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 787.238869] Lustre: 24330:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.250762] Lustre: 24330:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 787.257378] Lustre: 24330:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 787.269090] Lustre: 24330:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 787.277388] Lustre: 24330:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.588177] Lustre: 25570:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 790.614976] Lustre: 25570:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 790.644576] Lustre: 25570:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 790.659890] Lustre: 25570:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.694147] Lustre: 25570:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 790.711212] Lustre: 25570:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.718052] Lustre: 25570:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 790.745232] Lustre: 25570:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.756444] Lustre: 25570:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 790.764656] Lustre: 25570:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.772378] Lustre: 25570:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.778353] Lustre: 25570:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.188357] Lustre: 24329:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 799.204372] Lustre: 24329:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 799.212142] Lustre: 24329:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.221066] Lustre: 24329:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 799.237031] Lustre: 24329:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.248038] Lustre: 24329:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 799.262402] Lustre: 24329:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 799.273947] Lustre: 24329:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 799.283583] Lustre: 24329:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 799.291589] Lustre: 24329:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 799.298192] Lustre: 24329:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 799.305231] Lustre: 24329:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 810.405272] Lustre: 24331:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 810.435225] Lustre: 24331:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 810.449696] Lustre: 24331:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 810.456920] Lustre: 24331:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 810.467067] Lustre: 24331:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 810.478975] Lustre: 24331:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 810.492012] Lustre: 24331:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 810.504289] Lustre: 24331:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 810.512162] Lustre: 24331:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 810.522967] Lustre: 24331:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 810.534155] Lustre: 24331:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 810.542700] Lustre: 24331:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 858.596176] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.610705] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.634957] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 863.717861] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 868.319158] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 868.612082] Lustre: server umount lustre-OST07c6 complete [ 878.112475] Lustre: server umount lustre-MDT0000 complete [ 897.147900] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 900.933095] Key type lgssc unregistered [ 901.306524] LNet: 27851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.314635] LNetError: 27851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.341629] LNet: Removed LNI 192.168.201.149@tcp [ 902.444123] Key type .llcrypt unregistered [ 902.445723] Key type ._llcrypt unregistered [ 923.444991] Key type ._llcrypt registered [ 923.446939] Key type .llcrypt registered [ 923.594798] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 925.220672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.572205] alg: No test for adler32 (adler32-zlib) [ 926.954371] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 927.393296] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 929.151229] Key type lgssc registered [ 931.253817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 940.999375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.018807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.025449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.012371] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 953.314717] LustreError: 29112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.344554] LustreError: 29112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 958.204186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.434145] LustreError: 29111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.800461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 964.054467] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 970.890447] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 974.425152] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 982.441500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.775590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.897841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 990.536494] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 997.689668] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1013.729358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1013.729865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.734663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.768451] Lustre: Skipped 1 previous similar message [ 1015.601296] Lustre: server umount lustre-OST0000 complete [ 1020.386165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1020.405325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.422956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1020.437857] Lustre: Skipped 1 previous similar message [ 1023.976322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.976596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1024.039597] Lustre: Skipped 1 previous similar message [ 1026.074676] Lustre: server umount lustre-MDT0000 complete [ 1029.091171] LustreError: 29111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.125030] LustreError: 29111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1030.185163] LustreError: 29092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381473 with bad export cookie 12199391920371416966 [ 1030.204686] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.648930] Lustre: server umount lustre-MDT0001 complete [ 1043.374590] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1046.467834] Key type lgssc unregistered [ 1046.816970] LNet: 32491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.824366] LNetError: 32491:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.834885] LNet: Removed LNI 192.168.201.149@tcp [ 1047.745740] Key type .llcrypt unregistered [ 1047.748074] Key type ._llcrypt unregistered [ 1068.900889] Key type ._llcrypt registered [ 1068.903272] Key type .llcrypt registered [ 1069.049949] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1070.840186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.898481] alg: No test for adler32 (adler32-zlib) [ 1071.969634] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1072.207130] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1073.951204] Key type lgssc registered [ 1075.523719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1083.611503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1083.644418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.376203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.956308] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1095.650816] LustreError: 33744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.700920] LustreError: 33744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1100.665969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.776800] LustreError: 33745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1101.150869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.343766] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1111.994720] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1115.903830] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1123.138380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.614145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.814349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1131.519551] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1139.523187] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1151.461985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1151.462096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.470097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1151.494965] Lustre: Skipped 1 previous similar message [ 1156.596216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1156.605126] Lustre: Skipped 2 previous similar messages [ 1157.169941] Lustre: server umount lustre-OST0000 complete [ 1161.696815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.700369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.725336] Lustre: Skipped 1 previous similar message [ 1161.757259] Lustre: Skipped 1 previous similar message [ 1166.819415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.825499] Lustre: Skipped 1 previous similar message [ 1167.001489] Lustre: server umount lustre-MDT0000 complete [ 1171.356809] LustreError: 33724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381614 with bad export cookie 1623713279836014341 [ 1171.362881] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.539545] Lustre: server umount lustre-MDT0001 complete [ 1183.419158] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1186.474131] Key type lgssc unregistered [ 1186.778884] LNet: 36919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.792579] LNetError: 36919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.811587] LNet: Removed LNI 192.168.201.149@tcp [ 1187.597273] Key type .llcrypt unregistered [ 1187.599454] Key type ._llcrypt unregistered [ 1214.001818] Key type ._llcrypt registered [ 1214.003185] Key type .llcrypt registered [ 1214.068740] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1214.889097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1214.919059] alg: No test for adler32 (adler32-zlib) [ 1216.160971] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1216.470776] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1218.234682] Key type lgssc registered [ 1219.765893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1228.916883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1228.950606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.809836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.401713] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1240.038370] LustreError: 38326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.047679] LustreError: 38326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1245.154242] LustreError: 38327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.552493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.214222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1252.676792] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1259.730603] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1264.967935] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1273.217481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1273.813807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.848418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1281.054577] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1287.209850] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.377681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.385535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1297.399230] Lustre: Skipped 2 previous similar messages [ 1297.404766] Lustre: Skipped 1 previous similar message [ 1300.002741] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 1302.501299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.513175] Lustre: Skipped 2 previous similar messages [ 1303.324628] Lustre: server umount lustre-MDT0000 complete [ 1305.080371] LustreError: 38322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.300110] LustreError: 38308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381750 with bad export cookie 16556847769351976888 [ 1307.310400] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.316455] LustreError: 38308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1307.620877] LustreError: 38321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.628890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.651780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1312.739217] LustreError: 38322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1312.739604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1312.758221] LustreError: 38322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1312.773898] Lustre: Skipped 1 previous similar message [ 1313.884675] Lustre: server umount lustre-MDT0001 complete [ 1328.735505] Lustre: server umount lustre-OST0000 complete [ 1345.352927] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1347.898280] Key type lgssc unregistered [ 1348.219734] LNet: 41833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.232343] LNetError: 41833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.250603] LNet: Removed LNI 192.168.201.149@tcp [ 1349.279354] Key type .llcrypt unregistered [ 1349.280978] Key type ._llcrypt unregistered [ 1375.630675] Key type ._llcrypt registered [ 1375.633906] Key type .llcrypt registered [ 1375.835595] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1377.551699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.637634] alg: No test for adler32 (adler32-zlib) [ 1378.745694] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1378.960838] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1380.623150] Key type lgssc registered [ 1382.060861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1390.807411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1390.837052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.381855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.634737] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1401.830054] LustreError: 43234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.862969] LustreError: 43234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1405.689380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.126563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1410.845496] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1416.315886] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1420.142491] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1426.879967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.131333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.747221] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1437.389754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1438.983173] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1446.885421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1446.889225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.914175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1447.905092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1447.921860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1447.926224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.963556] Lustre: Skipped 2 previous similar messages [ 1451.938146] Lustre: server umount lustre-OST0000 complete [ 1472.997887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1473.014814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.042281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.909921] Lustre: server umount lustre-MDT0000 complete [ 1479.662746] LustreError: 44393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.696473] LustreError: 44393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1480.055173] LustreError: 43213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780381923 with bad export cookie 315274585387841961 [ 1480.060947] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.063971] LustreError: 43213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1486.481319] Lustre: server umount lustre-MDT0001 complete [ 1497.021707] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1499.891243] Key type lgssc unregistered [ 1500.162537] LNet: 46583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.183037] LNetError: 46583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.221896] LNet: Removed LNI 192.168.201.149@tcp [ 1502.082280] Key type .llcrypt unregistered [ 1502.087728] Key type ._llcrypt unregistered [ 1526.427667] Key type ._llcrypt registered [ 1526.431314] Key type .llcrypt registered [ 1526.566627] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1528.210492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1528.260167] alg: No test for adler32 (adler32-zlib) [ 1529.475881] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1529.755968] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1531.455236] Key type lgssc registered [ 1532.661037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.327762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1541.366940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.907858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.182925] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1553.378290] LustreError: 47991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.413560] LustreError: 47991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1555.860237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.208088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1561.871749] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1567.924186] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1571.927281] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1579.619302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1580.098547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.487798] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1588.277752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1591.502347] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.261731] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1600.309716] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1612.344318] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-02 2h34m12s [ 1613.808733] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1617.888117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1617.897375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.922715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.981836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.998157] Lustre: Skipped 4 previous similar messages [ 1619.983994] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 1621.950307] Lustre: server umount lustre-MDT0000 complete [ 1625.061620] LustreError: 47987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.087557] LustreError: 47987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1626.104911] LustreError: 47987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.585747] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1631.204459] LustreError: 47990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.216792] LustreError: 47990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1632.830063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.000928] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1633.238642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1633.281882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1634.285085] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1638.271386] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1642.159611] Lustre: DEBUG MARKER: Wait for df (36014) ... [ 1644.346764] Lustre: DEBUG MARKER: done [ 1652.704739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1652.715180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.722671] Lustre: Skipped 2 previous similar messages [ 1652.734896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1652.737451] Lustre: Skipped 1 previous similar message [ 1653.736282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1653.741356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.336514] Lustre: server umount lustre-OST0000 complete [ 1664.992418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.000297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1665.002500] Lustre: Skipped 1 previous similar message [ 1665.002828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1665.008067] Lustre: Skipped 4 previous similar messages [ 1675.231763] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1675.474905] Lustre: server umount lustre-MDT0000 complete [ 1678.305811] LustreError: 47987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.339038] LustreError: 47987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1679.396628] LustreError: 47971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382122 with bad export cookie 9978271867327743570 [ 1679.414541] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.683751] Lustre: server umount lustre-MDT0001 complete [ 1690.922224] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1694.073315] Key type lgssc unregistered [ 1694.460573] LNet: 52539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.470612] LNetError: 52539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.529302] LNet: Removed LNI 192.168.201.149@tcp [ 1696.503110] Key type .llcrypt unregistered [ 1696.505313] Key type ._llcrypt unregistered [ 1734.709910] Key type ._llcrypt registered [ 1734.716473] Key type .llcrypt registered [ 1734.836720] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1736.352213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1736.415533] alg: No test for adler32 (adler32-zlib) [ 1737.590420] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1737.834903] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1739.487175] Key type lgssc registered [ 1740.456920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1746.709867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1746.725679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.174616] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1748.205670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1748.423194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.875326] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1763.544980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.640997] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1763.664797] Lustre: 54734:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1763.986314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1768.371146] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1773.383431] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1777.461496] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1785.293903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1785.579297] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1785.807959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.668031] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1793.841788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1799.404817] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1806.104294] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-02 2h37m26s [ 1807.745079] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1816.056745] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1818.617273] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1818.642848] LustreError: 54741:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1839.152107] Lustre: lustre-MDT0000: Client 8f3d6e85-e0f8-4f7b-be1f-7aacbdb2db5a (at 192.168.201.49@tcp) reconnecting [ 1858.703402] LustreError: 54741:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1861.100487] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1863.601238] Lustre: DEBUG MARKER: done [ 1870.815886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1870.817846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.833966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1870.839253] Lustre: Skipped 1 previous similar message [ 1870.847794] Lustre: Skipped 1 previous similar message [ 1872.292962] Lustre: server umount lustre-OST0000 complete [ 1875.939865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.946434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1875.970824] Lustre: Skipped 1 previous similar message [ 1877.026187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.031460] Lustre: Skipped 2 previous similar messages [ 1881.060803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1881.075106] Lustre: Skipped 1 previous similar message [ 1882.113591] Lustre: server umount lustre-MDT0000 complete [ 1887.227919] LustreError: 54728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382330 with bad export cookie 18146427775546917734 [ 1887.244750] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1887.246051] LustreError: 54728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1887.948719] Lustre: server umount lustre-MDT0001 complete [ 1901.775645] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1904.590792] Key type lgssc unregistered [ 1904.843847] LNet: 58707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.849290] LNetError: 58707:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.866183] LNet: Removed LNI 192.168.201.149@tcp [ 1905.624980] Key type .llcrypt unregistered [ 1905.626695] Key type ._llcrypt unregistered [ 1952.896844] Key type ._llcrypt registered [ 1952.903741] Key type .llcrypt registered [ 1953.058580] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1954.994870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.017451] alg: No test for adler32 (adler32-zlib) [ 1956.199935] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 1956.413112] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1958.151189] Key type lgssc registered [ 1959.392345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1963.375913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1963.417725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.848597] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1964.885442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1965.108455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1966.134117] Lustre: Failing over lustre-MDT0000 [ 1966.745983] Lustre: server umount lustre-MDT0000 complete [ 1974.166485] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:40:15 (1780382415) [ 1981.090715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.494743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.115301] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1998.021617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.407563] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1998.478289] Lustre: 61492:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1999.031028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2008.017491] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2015.172412] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2019.202482] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2026.349280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2026.639794] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2026.821372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.474077] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2032.904156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2038.923611] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2045.093843] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2050.777080] LustreError: 60413:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000070eb3080 x1866866254450816/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2053.606338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.610048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.624561] Lustre: Skipped 2 previous similar messages [ 2053.632451] Lustre: Skipped 2 previous similar messages [ 2058.724738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.730939] Lustre: Skipped 2 previous similar messages [ 2063.840162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2063.847787] Lustre: Skipped 1 previous similar message [ 2065.375289] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2066.074913] Lustre: server umount lustre-MDT0000 complete [ 2070.924394] LustreError: 61485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382514 with bad export cookie 4205737373199007454 [ 2070.964390] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.125316] Lustre: server umount lustre-MDT0001 complete [ 2077.202830] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2082.803289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2090.465177] Lustre: 60414:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780382517/real 1780382517] req@00000000ae1b778e x1866866254456448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780382533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.514456] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.487371] Lustre: 60413:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780382518/real 1780382518] req@0000000075c529bb x1866866254456576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780382534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2093.937723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.589067] Lustre: 60415:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780382522/real 1780382522] req@00000000e706b6b9 x1866866254456704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780382538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2096.223136] LustreError: 64911:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2096.232099] LustreError: 64911:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000008d897079 x1866866254456832/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1780382539 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2096.252085] LustreError: 64911:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2096.676919] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3a5dc7dba41c87c1 [ 2096.686709] Lustre: MGC192.168.201.149@tcp: Connection restored to 0@lo (at 0@lo) [ 2096.983059] LustreError: 63114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.999923] LustreError: 63114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2097.120664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.263141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2098.080052] Lustre: 60416:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780382524/real 1780382524] req@0000000087a5ded2 x1866866254456960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780382540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2099.172511] LustreError: 63115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.188232] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2099.200600] LustreError: 63115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2100.265622] LustreError: 64926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.289904] LustreError: 64926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2103.226681] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2103.263208] Lustre: 60414:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780382529/real 1780382529] req@0000000080e8c758 x1866866254457088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780382545 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2105.327809] LustreError: 64924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.358826] LustreError: 64924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2110.436420] LustreError: 63114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.463981] LustreError: 63114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2115.448378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.963476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2121.226060] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2122.710878] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2129.229625] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2134.441737] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2141.666392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.671830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2141.674381] Lustre: Skipped 2 previous similar messages [ 2141.704826] Lustre: Skipped 2 previous similar messages [ 2141.881360] Lustre: server umount lustre-MDT0000 complete [ 2146.254750] LustreError: 61485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382589 with bad export cookie 4205737373199009729 [ 2146.264250] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2146.281536] LustreError: 61485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2146.786229] LustreError: 65269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.791715] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.810597] LustreError: 65269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2146.901335] Lustre: server umount lustre-MDT0001 complete [ 2153.745846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2165.243469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.361409] LustreError: 60412:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007533ceab x1866866254483712/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2171.998226] LustreError: 63115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.180252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.256639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2173.228121] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2177.340752] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2185.995612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.612530] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2191.863502] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2198.600761] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2203.244530] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2209.327062] Lustre: server umount lustre-MDT0000 complete [ 2212.336761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.345758] LustreError: 63114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.388710] Lustre: Skipped 2 previous similar messages [ 2212.391754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2212.424423] LustreError: 63114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2212.899472] LustreError: 61485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382656 with bad export cookie 4205737373199011108 [ 2212.908629] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2212.911486] LustreError: 61485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2213.452325] Lustre: server umount lustre-MDT0001 complete [ 2220.037052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2226.881115] Lustre: DEBUG MARKER: files compared the same [ 2240.997299] Lustre: server umount lustre-OST0000 complete [ 2264.285117] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2267.798696] Key type lgssc unregistered [ 2268.252819] LNet: 70417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.270343] LNetError: 70417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.289127] LNet: Removed LNI 192.168.201.149@tcp [ 2270.538079] Key type .llcrypt unregistered [ 2270.543386] Key type ._llcrypt unregistered [ 2300.955878] Key type ._llcrypt registered [ 2300.957482] Key type .llcrypt registered [ 2301.069897] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2301.886632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.087643] alg: No test for adler32 (adler32-zlib) [ 2303.394718] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2303.716129] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2305.375301] Key type lgssc registered [ 2306.295415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2315.458035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2315.476551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.168368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.813623] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 2327.523430] LustreError: 71821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.546116] LustreError: 71821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2330.622179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.120670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.525089] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 2342.891524] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2347.542429] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2355.405467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2355.769789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.903447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2363.014247] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 2369.186840] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2382.303933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2382.307716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.322437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.328471] Lustre: Skipped 1 previous similar message [ 2385.194699] Lustre: server umount lustre-OST0000 complete [ 2392.544764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.546889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2392.562581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2392.580375] Lustre: Skipped 2 previous similar messages [ 2396.005588] Lustre: server umount lustre-MDT0000 complete [ 2397.663699] LustreError: 71817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.686249] LustreError: 71817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2400.013757] LustreError: 71801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780382843 with bad export cookie 6036392237304262342 [ 2400.016903] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2400.019224] LustreError: 71801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2400.541776] Lustre: server umount lustre-MDT0001 complete [ 2413.540260] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2416.952528] Key type lgssc unregistered [ 2417.422592] LNet: 74997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2417.430943] LNetError: 74997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2417.462478] LNet: Removed LNI 192.168.201.149@tcp [ 2418.463691] Key type .llcrypt unregistered [ 2418.466471] Key type ._llcrypt unregistered [ 2447.444842] Key type ._llcrypt registered [ 2447.446830] Key type .llcrypt registered [ 2447.544151] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2448.463175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2448.484680] alg: No test for adler32 (adler32-zlib) [ 2449.749973] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2450.110366] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2451.967174] Key type lgssc registered [ 2453.294147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2463.470707] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2463.503442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.202718] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2484.319264] LustreError: 76371:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.149@tcp: failed processing log, type 4: rc = -110 [ 2511.007250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.377552] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2523.411489] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2533.327825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.496752] Lustre: *** cfs_fail_loc=706, val=0*** [ 2533.975602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.082715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2538.980558] LustreError: 76397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.001485] LustreError: 76397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2539.478869] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2544.102861] LustreError: 76396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.170506] LustreError: 76397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.185874] LustreError: 76397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2550.245631] LustreError: 77343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.286338] LustreError: 77343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2550.816438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.171977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2556.898931] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2563.844434] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2567.927344] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2571.744048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2571.761410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.769399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.787587] Lustre: Skipped 1 previous similar message [ 2571.804459] Lustre: Skipped 1 previous similar message [ 2576.872762] LustreError: 76385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.898048] LustreError: 76385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2577.011759] Lustre: server umount lustre-OST0000 complete [ 2581.988718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.991486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2581.998907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2582.010413] Lustre: Skipped 1 previous similar message [ 2587.118691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.123459] Lustre: Skipped 2 previous similar messages [ 2592.227640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.236018] Lustre: Skipped 1 previous similar message [ 2596.319165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2596.504397] Lustre: server umount lustre-MDT0000 complete [ 2597.354914] LustreError: 77346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.370280] LustreError: 77346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2600.678250] LustreError: 76378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383043 with bad export cookie 6371411156454950487 [ 2600.694964] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2601.120840] Lustre: server umount lustre-MDT0001 complete [ 2613.547357] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2617.399065] Key type lgssc unregistered [ 2617.731178] LNet: 79631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2617.750081] LNetError: 79631:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2617.769774] LNet: Removed LNI 192.168.201.149@tcp [ 2618.918328] Key type .llcrypt unregistered [ 2618.924324] Key type ._llcrypt unregistered [ 2647.896932] Key type ._llcrypt registered [ 2647.898577] Key type .llcrypt registered [ 2647.980244] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2649.398236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2649.589033] alg: No test for adler32 (adler32-zlib) [ 2650.748841] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 2651.108686] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2652.768164] Key type lgssc registered [ 2654.102578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2664.000405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2664.034672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.005605] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2680.099607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.945088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2685.923315] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.967053] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2686.120619] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2691.042597] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.064982] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2696.073880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.162234] LustreError: 81704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.189823] LustreError: 81704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2696.578290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.794416] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.805234] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2701.982815] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2705.900297] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.911579] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2711.504703] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2711.512970] Lustre: MGS: Client 2063c451-20f9-44da-98c2-4ec2804f9a7b (at 0@lo) reconnecting [ 2711.535316] LustreError: 81715:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2711.730841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.832747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2716.324448] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2732.512763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.516688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.522532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.523352] Lustre: Skipped 1 previous similar message [ 2732.530732] Lustre: Skipped 1 previous similar message [ 2737.634695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.641788] Lustre: Skipped 1 previous similar message [ 2738.568701] Lustre: server umount lustre-OST0000 complete [ 2742.752660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2742.753806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.776816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2742.781449] Lustre: Skipped 1 previous similar message [ 2747.872952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.887130] Lustre: Skipped 1 previous similar message [ 2748.986815] Lustre: server umount lustre-MDT0000 complete [ 2752.997153] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.017390] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2754.242177] Lustre: server umount lustre-MDT0001 complete [ 2758.065346] Lustre: server umount MGS complete [ 2769.179423] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:53:30 (1780383210) [ 2807.753500] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2819.435467] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2864.123257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.794579] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2879.374699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.704139] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2879.756837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2879.864238] Lustre: lustre-MDT0001: new disk, initializing [ 2879.949641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2880.008526] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2880.012869] Lustre: 88793:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2880.039260] LustreError: 88793:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2884.361107] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2897.774097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.089427] Lustre: lustre-OST0000: new disk, initializing [ 2898.093788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2898.103394] Lustre: 89542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2898.197726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.714611] Lustre: 89781:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000004ca7f4e1 x1866866983214208/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780383364 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2907.094986] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2916.368142] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2924.772553] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2924.792957] Lustre: MGS: Client 7f0af11b-abdf-4b21-8e92-91e130e925cf (at 0@lo) reconnecting [ 2924.870961] Lustre: 88080:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2924.896137] LustreError: 88779:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2924.919244] LustreError: 88779:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2924.982183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2925.204475] Lustre: lustre-MDT0000: new disk, initializing [ 2925.391475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2925.418323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2925.459359] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2926.496439] Lustre: 90500:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000b74c9f52 x1866866983221376/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780383385 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2931.073839] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2931.679901] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2935.598349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2935.608975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2935.758526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2959.330748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.338978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2959.361996] Lustre: Skipped 2 previous similar messages [ 2964.926908] Lustre: server umount lustre-OST0000 complete [ 2969.570200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.575325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.582832] Lustre: Skipped 1 previous similar message [ 2969.599264] Lustre: Skipped 4 previous similar messages [ 2974.622624] Lustre: server umount lustre-MDT0000 complete [ 2974.688834] LustreError: 88764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.743079] LustreError: 88764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2985.728048] Lustre: server umount lustre-MDT0001 complete [ 2989.191437] Lustre: server umount MGS complete [ 3001.707701] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:57:22 (1780383442) [ 3022.237971] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3025.164574] Key type lgssc unregistered [ 3025.500774] LNet: 92966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3025.513098] LNetError: 92966:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3025.531586] LNet: Removed LNI 192.168.201.149@tcp [ 3026.704853] Key type .llcrypt unregistered [ 3026.706919] Key type ._llcrypt unregistered [ 3051.868446] Key type ._llcrypt registered [ 3051.872927] Key type .llcrypt registered [ 3051.996089] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3053.461356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3053.598832] alg: No test for adler32 (adler32-zlib) [ 3054.712710] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3055.100424] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3056.761155] Key type lgssc registered [ 3058.276481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3067.667725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3067.694784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.432704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3074.615954] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3079.659272] LustreError: 94175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.677802] LustreError: 94175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3084.770802] LustreError: 94176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.419467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.034237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3091.006704] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3097.357511] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3101.764889] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3111.277944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3111.707894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.825393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3119.511959] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3126.650295] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3140.579408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3140.587490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.601327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3141.600321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3141.603744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.609121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.692317] Lustre: server umount lustre-OST0000 complete [ 3147.748642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3147.762383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.775376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.782577] Lustre: Skipped 1 previous similar message [ 3151.840167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3151.841974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.960461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.966495] Lustre: Skipped 2 previous similar messages [ 3162.080219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3162.392519] Lustre: server umount lustre-MDT0000 complete [ 3166.185017] LustreError: 94176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.214357] LustreError: 94176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3166.743534] LustreError: 94155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383609 with bad export cookie 13982863678153614074 [ 3166.752216] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3167.181688] Lustre: server umount lustre-MDT0001 complete [ 3180.270981] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3183.599756] Key type lgssc unregistered [ 3183.953584] LNet: 97356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3183.964158] LNetError: 97356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3183.984851] LNet: Removed LNI 192.168.201.149@tcp [ 3184.963406] Key type .llcrypt unregistered [ 3184.965732] Key type ._llcrypt unregistered [ 3203.043430] Key type ._llcrypt registered [ 3203.047476] Key type .llcrypt registered [ 3203.176088] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3204.596380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3204.795951] alg: No test for adler32 (adler32-zlib) [ 3206.108710] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3206.405225] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3208.167283] Key type lgssc registered [ 3209.566974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3217.178402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3217.190106] LustreError: 98417:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3217.236541] LustreError: 98424:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3217.247473] LustreError: 98417:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4956 [ 3217.255060] LustreError: 98417:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3217.267410] LustreError: 98417:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3217.280671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.884332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.028036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.279666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3232.390419] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3238.409531] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing lsmod [ 3241.305437] LustreError: 99602:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3241.362643] LustreError: 99609:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3241.371215] LustreError: 99602:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4949 [ 3241.374432] LustreError: 99602:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3241.392288] LustreError: 99602:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3241.403865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3241.905874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.959812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3248.098783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3248.104645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.123830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3249.123705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.131894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3249.134660] Lustre: Skipped 1 previous similar message [ 3252.261137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.264725] Lustre: Skipped 2 previous similar messages [ 3257.318447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3257.327402] Lustre: Skipped 2 previous similar messages [ 3259.359120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3259.607210] Lustre: server umount lustre-MDT0000 complete [ 3262.435701] LustreError: 98471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.464190] LustreError: 98471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3264.011520] LustreError: 98439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383707 with bad export cookie 17342888932667497569 [ 3264.016966] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.034581] LustreError: 98439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3264.631303] LustreError: 98906:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e7e0e6af x1866867565157248/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3264.868351] Lustre: server umount lustre-MDT0001 complete [ 3279.939452] Lustre: server umount lustre-OST0000 complete [ 3292.199106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.887794] LustreError: 100676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.957854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.957305] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3298.279922] LustreError: 100677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.392362] LustreError: 100676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.008504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3309.348600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3315.270217] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3320.937684] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3325.553099] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3334.357037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3334.980137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3339.123781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3344.329279] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3349.877419] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3362.787335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3362.789994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.803546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.834782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3364.851567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.722796] Lustre: server umount lustre-OST0000 complete [ 3375.087801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.089352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.111537] Lustre: Skipped 1 previous similar message [ 3375.127560] Lustre: Skipped 3 previous similar messages [ 3385.823219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3385.958646] Lustre: server umount lustre-MDT0000 complete [ 3390.393440] LustreError: 100658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383833 with bad export cookie 17342888932667498500 [ 3390.418651] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3390.438581] LustreError: 100658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3390.996680] Lustre: server umount lustre-MDT0001 complete [ 3403.014914] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3408.166649] Key type lgssc unregistered [ 3408.540738] LNet: 103856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3408.549888] LNetError: 103856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3408.574949] LNet: Removed LNI 192.168.201.149@tcp [ 3409.464911] Key type .llcrypt unregistered [ 3409.466312] Key type ._llcrypt unregistered [ 3435.638263] Key type ._llcrypt registered [ 3435.641422] Key type .llcrypt registered [ 3435.743578] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3437.044377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3437.058195] alg: No test for adler32 (adler32-zlib) [ 3438.087836] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3438.287452] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3439.999528] Key type lgssc registered [ 3441.143247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3450.121667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3450.141408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.783450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.747822] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3461.102884] LustreError: 105259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.117623] LustreError: 105259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3466.232561] LustreError: 105260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.739894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3468.092195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3471.950599] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3477.090926] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3481.093649] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3488.098058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3488.372849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.677409] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3496.566741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3500.369385] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3506.302533] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3510.297498] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3514.359836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3514.366787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.391843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3516.119492] Lustre: server umount lustre-OST0000 complete [ 3522.545613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.567881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.578602] Lustre: Skipped 1 previous similar message [ 3522.604700] Lustre: Skipped 1 previous similar message [ 3527.652118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3527.657530] Lustre: Skipped 1 previous similar message [ 3530.820087] Lustre: server umount lustre-MDT0000 complete [ 3532.770198] LustreError: 107411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.810608] LustreError: 107411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3535.172974] LustreError: 107452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780383978 with bad export cookie 9416089188666934722 [ 3535.177630] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3535.185039] LustreError: 107452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3535.662056] Lustre: server umount lustre-MDT0001 complete [ 3547.727600] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3550.443648] Key type lgssc unregistered [ 3550.708777] LNet: 108541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3550.717275] LNetError: 108541:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3550.734318] LNet: Removed LNI 192.168.201.149@tcp [ 3551.620270] Key type .llcrypt unregistered [ 3551.622758] Key type ._llcrypt unregistered [ 3571.421656] Key type ._llcrypt registered [ 3571.424199] Key type .llcrypt registered [ 3571.530440] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3572.790202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3572.895196] alg: No test for adler32 (adler32-zlib) [ 3574.302511] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3574.653913] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3576.323173] Key type lgssc registered [ 3578.772029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3590.175532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3590.197538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.980421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.705145] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3602.420328] LustreError: 109747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.443266] LustreError: 109747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3607.053200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.446677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3614.210660] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3621.659589] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3628.070615] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3636.449479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.226221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3638.269218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3645.722185] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3652.443687] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3662.823338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3662.825036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.842521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3662.859935] Lustre: Skipped 1 previous similar message [ 3667.444372] Lustre: server umount lustre-OST0000 complete [ 3673.056628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.062166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3673.081432] Lustre: Skipped 1 previous similar message [ 3673.096590] Lustre: Skipped 2 previous similar messages [ 3678.176225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.298561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.304104] Lustre: Skipped 2 previous similar messages [ 3685.343877] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3685.840422] Lustre: server umount lustre-MDT0000 complete [ 3688.423330] LustreError: 109746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.455042] LustreError: 109746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3690.867309] LustreError: 109728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384134 with bad export cookie 1926591357620032417 [ 3690.888096] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3690.899254] LustreError: 109728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3691.489323] Lustre: server umount lustre-MDT0001 complete [ 3707.893318] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3711.370610] Key type lgssc unregistered [ 3711.645645] LNet: 112927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3711.649873] LNetError: 112927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3711.669287] LNet: Removed LNI 192.168.201.149@tcp [ 3712.518224] Key type .llcrypt unregistered [ 3712.519921] Key type ._llcrypt unregistered [ 3739.296831] Key type ._llcrypt registered [ 3739.300226] Key type .llcrypt registered [ 3739.445635] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3741.163102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3741.260973] alg: No test for adler32 (adler32-zlib) [ 3742.393414] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3742.827445] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3744.615160] Key type lgssc registered [ 3745.788412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3753.494618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3753.524159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.108607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.879861] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3765.738557] LustreError: 114333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3765.762124] LustreError: 114333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3769.501785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.115308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3774.711583] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3780.977685] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3785.300191] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3793.422697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3793.871686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3800.068299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3800.110279] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3805.735372] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3812.715401] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3817.647118] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3817.659109] Lustre: Skipped 2 previous similar messages [ 3822.687817] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3822.700785] Lustre: Skipped 2 previous similar messages [ 3822.701473] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3822.741929] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3822.754612] Lustre: Skipped 1 previous similar message [ 3831.865228] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3837.536502] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.49@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3837.543808] Lustre: Skipped 2 previous similar messages [ 3839.511671] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.49@tcp 192.168.0.[2,10]@tcp [ 3848.499457] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.49/24@tcp in log lustre-MDT0000 [ 3848.517474] Lustre: Skipped 2 previous similar messages [ 3854.444224] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.49/24@tcp [ 3854.468141] Lustre: Skipped 1 previous similar message [ 3861.474958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3861.482864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.484914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3861.509905] Lustre: Skipped 1 previous similar message [ 3861.546949] Lustre: Skipped 1 previous similar message [ 3863.486502] Lustre: server umount lustre-OST0000 complete [ 3871.712709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.741178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.762141] Lustre: Skipped 1 previous similar message [ 3874.346887] Lustre: server umount lustre-MDT0000 complete [ 3876.832726] LustreError: 116056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.859786] LustreError: 116056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3879.446924] LustreError: 114313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384322 with bad export cookie 7314058678473045230 [ 3879.449597] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3879.466657] LustreError: 114313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3880.055799] Lustre: server umount lustre-MDT0001 complete [ 3896.095786] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3899.668551] Key type lgssc unregistered [ 3900.053191] LNet: 118968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3900.065396] LNetError: 118968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3900.090195] LNet: Removed LNI 192.168.201.149@tcp [ 3901.033950] Key type .llcrypt unregistered [ 3901.035810] Key type ._llcrypt unregistered [ 3935.928193] Key type ._llcrypt registered [ 3935.932588] Key type .llcrypt registered [ 3936.143895] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3938.068767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3938.372981] alg: No test for adler32 (adler32-zlib) [ 3939.549832] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 3939.949518] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3941.767962] Key type lgssc registered [ 3943.587781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3952.741871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3952.759212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3954.515576] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3954.552706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3954.571819] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.49/24@tcp [ 3959.962340] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3963.882638] LustreError: 120519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.914946] LustreError: 120519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3968.998570] LustreError: 120518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.488464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.967587] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3971.015168] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.49/24@tcp [ 3971.070852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3976.263567] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3983.516539] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3988.242657] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3995.645727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3995.973882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.166724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4002.374786] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4009.123212] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4022.756098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4022.771717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.786451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.814508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.816715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4024.820263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.354147] Lustre: server umount lustre-OST0000 complete [ 4032.482566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4032.488504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.512301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4032.522769] Lustre: Skipped 1 previous similar message [ 4035.044504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4035.045891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.737558] Lustre: server umount lustre-MDT0000 complete [ 4040.174925] LustreError: 121888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.219603] LustreError: 121888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4040.977671] LustreError: 120498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384484 with bad export cookie 17237502046497009342 [ 4040.987597] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.994995] LustreError: 120498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4041.454371] Lustre: server umount lustre-MDT0001 complete [ 4055.152776] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4058.531575] Key type lgssc unregistered [ 4058.968779] LNet: 123750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4058.979972] LNetError: 123750:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4060.005711] LNet: Removed LNI 192.168.201.149@tcp [ 4060.990190] Key type .llcrypt unregistered [ 4060.993742] Key type ._llcrypt unregistered [ 4090.351878] Key type ._llcrypt registered [ 4090.353485] Key type .llcrypt registered [ 4090.477704] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4107.103629] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4108.027474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4108.048811] alg: No test for adler32 (adler32-zlib) [ 4109.387097] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4109.668869] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4111.327702] Key type lgssc registered [ 4112.510818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4149.320976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4149.347745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.809675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4150.838430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4150.984393] Lustre: lustre-MDT0000: new disk, initializing [ 4151.130209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.168673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4156.527721] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4169.240821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4169.429399] Lustre: 127124:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4169.513928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4169.519308] Lustre: Skipped 1 previous similar message [ 4169.605993] Lustre: lustre-MDT0001: new disk, initializing [ 4169.698466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4169.733979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4169.745828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4174.588300] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4183.259346] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4187.484811] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4194.584107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4194.923061] Lustre: lustre-OST0000: new disk, initializing [ 4194.929649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4194.938719] Lustre: 128867:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4195.076535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.688282] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4204.642200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4204.662170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4204.874403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4212.713478] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4217.545936] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4217.852326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4223.045816] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4223.304034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4230.639385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4230.656633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.670574] LustreError: Skipped 1 previous similar message [ 4230.675434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4230.692865] Lustre: Skipped 1 previous similar message [ 4230.722724] Lustre: Skipped 1 previous similar message [ 4231.962923] Lustre: server umount lustre-OST0000 complete [ 4236.256778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4236.272594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.291101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4240.866653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4240.869194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.550303] Lustre: server umount lustre-MDT0000 complete [ 4246.856226] LustreError: 128866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384690 with bad export cookie 11732435224819505957 [ 4246.868827] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4247.283365] Lustre: server umount lustre-MDT0001 complete [ 4256.815531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4257.456910] LustreError: 130939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4257.481187] LustreError: 130939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4257.574516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.868532] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4262.884700] LustreError: 130940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.979928] LustreError: 130939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4272.103407] LustreError: 130940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4272.508084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.993062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4278.014414] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4284.841810] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4288.801588] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4296.133757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4296.498168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.562290] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4310.355205] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4316.046622] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4316.279740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4321.332460] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4321.525877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4331.437269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4331.611451] Lustre: lustre-OST0001: new disk, initializing [ 4331.613878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4331.619647] Lustre: 133935:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4331.700505] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4337.301877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4337.314172] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4337.458455] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4338.854498] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4348.061721] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4348.329496] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4353.565916] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4353.795450] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4358.168299] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff957c055dd000.ost_server_uuid 50 [ 4359.756295] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff957c055dd000.ost_server_uuid in FULL state after 0 sec [ 4370.631936] Lustre: server umount lustre-OST0001 complete [ 4373.484401] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4373.485275] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.510923] LustreError: Skipped 1 previous similar message [ 4373.513593] LustreError: 134868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.581214] Lustre: Skipped 1 previous similar message [ 4375.525372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4375.537348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4375.543564] Lustre: Skipped 1 previous similar message [ 4378.609244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.618194] Lustre: Skipped 1 previous similar message [ 4380.934455] Lustre: server umount lustre-OST0000 complete [ 4385.760534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4385.765300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4385.781492] Lustre: Skipped 2 previous similar messages [ 4388.837617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.852341] Lustre: Skipped 1 previous similar message [ 4391.750212] Lustre: server umount lustre-MDT0000 complete [ 4393.955264] LustreError: 130939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.975374] LustreError: 130939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4396.361331] LustreError: 135092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780384839 with bad export cookie 11732435224819506839 [ 4396.365189] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4396.366366] LustreError: 135092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4396.869593] Lustre: server umount lustre-MDT0001 complete [ 4417.534578] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4420.683605] Key type lgssc unregistered [ 4421.111430] LNet: 136776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4421.122296] LNetError: 136776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4421.151836] LNet: Removed LNI 192.168.201.149@tcp [ 4422.323665] Key type .llcrypt unregistered [ 4422.325993] Key type ._llcrypt unregistered [ 4463.588428] Key type ._llcrypt registered [ 4463.593959] Key type .llcrypt registered [ 4463.704916] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4480.039208] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4481.782117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4481.812124] alg: No test for adler32 (adler32-zlib) [ 4482.836976] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4483.070262] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4484.729120] Key type lgssc registered [ 4486.508402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4524.706685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4524.738479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.197576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4526.232286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4526.325940] Lustre: lustre-MDT0000: new disk, initializing [ 4526.419457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.447425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4531.659543] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4545.251876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.481567] Lustre: 140899:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4545.554936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4545.558848] Lustre: Skipped 1 previous similar message [ 4545.688718] Lustre: lustre-MDT0001: new disk, initializing [ 4545.817345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4545.910621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4545.964879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4550.408470] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4558.894039] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4563.670892] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4570.938047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4571.381866] Lustre: lustre-OST0000: new disk, initializing [ 4571.400772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4571.412689] Lustre: 142639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4571.532264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.904940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4578.921108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4579.139231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4580.068828] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4588.790395] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4592.929876] Lustre: Failing over lustre-OST0000 [ 4593.046381] Lustre: server umount lustre-OST0000 complete [ 4611.039207] Lustre: 139123:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780385037/real 1780385037] req@00000000350b48e3 x1866868904200320/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780385053 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4611.039385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.066321] Lustre: 139123:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4611.110778] Lustre: Skipped 1 previous similar message [ 4613.218151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.597310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.609203] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4615.649149] Lustre: 139121:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780385042/real 1780385042] req@0000000089a6b897 x1866868904201600/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780385058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4615.684173] Lustre: 139121:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4620.043102] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4621.728109] Lustre: 139124:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780385048/real 1780385048] req@00000000a3d9ee47 x1866868904202752/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780385064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4621.750381] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4621.764135] Lustre: 139124:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4622.381363] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4622.383105] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4623.841912] Lustre: 139121:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780385048/real 1780385048] req@000000004e43494d x1866868904202624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780385064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4624.036176] Lustre: Failing over lustre-MDT0000 [ 4624.294130] Lustre: server umount lustre-MDT0000 complete [ 4626.915672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.922867] Lustre: Skipped 2 previous similar messages [ 4632.033211] LustreError: 140910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.050606] LustreError: 140910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4632.566174] LustreError: 140907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.152785] LustreError: 142647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.198271] LustreError: 142647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4642.272351] LustreError: 140910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.294374] LustreError: 140910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4643.295906] Lustre: 139123:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780385070/real 1780385070] req@00000000c8fbf54a x1866868904213632/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780385086 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4643.315672] Lustre: 139123:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4643.324795] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4644.080748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.930708] LustreError: 142647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.951106] LustreError: 142647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4653.559965] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd8a00c76349bb7e6 [ 4653.568454] Lustre: MGC192.168.201.149@tcp: Connection restored to 0@lo (at 0@lo) [ 4653.571215] Lustre: Skipped 1 previous similar message [ 4653.888441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.990649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4658.137516] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4658.168769] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4659.177765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4659.196307] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4659.250213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4668.905328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4668.917367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.928741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4674.551533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4674.564495] Lustre: Skipped 3 previous similar messages [ 4679.654274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4679.658949] Lustre: Skipped 1 previous similar message [ 4680.159144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4680.322395] Lustre: server umount lustre-OST0000 complete [ 4684.771665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.779768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4684.781363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.807448] Lustre: Skipped 2 previous similar messages [ 4689.890229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.899354] Lustre: Skipped 2 previous similar messages [ 4690.862580] Lustre: server umount lustre-MDT0000 complete [ 4693.750898] LustreError: 140892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385136 with bad export cookie 15609490010294433766 [ 4693.755328] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.761323] LustreError: 140892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4694.098154] Lustre: server umount lustre-MDT0001 complete [ 4705.249649] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4708.136527] Key type lgssc unregistered [ 4708.419899] LNet: 146134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4708.428524] LNetError: 146134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4709.476779] LNet: Removed LNI 192.168.201.149@tcp [ 4710.399232] Key type .llcrypt unregistered [ 4710.401171] Key type ._llcrypt unregistered [ 4746.302281] Key type ._llcrypt registered [ 4746.308162] Key type .llcrypt registered [ 4746.453735] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4747.932185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4748.047988] alg: No test for adler32 (adler32-zlib) [ 4749.118620] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 4749.385767] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4751.183208] Key type lgssc registered [ 4752.436782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4760.966176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4760.978794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.457852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.244892] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4771.809219] LustreError: 147637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4771.842306] LustreError: 147637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4772.571789] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4776.256276] LustreError: 147666:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4776.266977] LustreError: 147666:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4776.273921] LustreError: 147666:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4776.555636] Lustre: server umount lustre-MDT0000 complete [ 4791.961626] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4801.805854] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4835.658619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.843943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4835.874032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4835.943770] Lustre: lustre-MDT0000: new disk, initializing [ 4836.020542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4836.034427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4840.854517] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4850.920581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.049852] Lustre: 150823:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4851.099575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4851.103854] Lustre: Skipped 1 previous similar message [ 4851.174916] Lustre: lustre-MDT0001: new disk, initializing [ 4851.272177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4851.315097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4851.327662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4855.662512] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4862.554525] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4866.084793] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4872.172165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4872.391742] Lustre: lustre-OST0000: new disk, initializing [ 4872.397268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4872.404678] Lustre: 152561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4872.487829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4878.426211] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4878.461271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4878.479506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4878.643227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4885.244217] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4889.861714] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4891.663127] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:28:52 (1780385332) [ 5150.176093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.179224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5150.202919] Lustre: Skipped 1 previous similar message [ 5150.234660] Lustre: Skipped 2 previous similar messages [ 5155.303554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5155.318153] Lustre: Skipped 2 previous similar messages [ 5155.637241] Lustre: server umount lustre-MDT0000 complete [ 5159.420251] LustreError: 150814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385602 with bad export cookie 2558266382593552382 [ 5159.421133] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5159.427800] LustreError: 150814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5160.416035] LustreError: 150829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.426223] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.440857] Lustre: Skipped 1 previous similar message [ 5160.444821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5164.449479] LustreError: 150830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.450210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5165.896470] Lustre: server umount lustre-MDT0001 complete [ 5176.799204] Lustre: server umount lustre-OST0000 complete [ 5186.950192] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 5198.213157] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5238.312182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.624535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5238.656150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5238.749970] Lustre: lustre-MDT0000: new disk, initializing [ 5238.914182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5238.944507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5243.725914] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5256.036727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5256.213792] Lustre: 157081:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5256.327305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5256.334259] Lustre: Skipped 1 previous similar message [ 5256.499409] Lustre: lustre-MDT0001: new disk, initializing [ 5256.696839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5256.773463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5256.794600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5263.329093] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5272.382358] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5276.086284] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5282.651103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5283.058420] Lustre: lustre-OST0000: new disk, initializing [ 5283.062622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5283.071987] Lustre: 158823:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5283.164913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.513918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5288.519087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5288.651236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5289.577320] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5296.455052] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5301.370334] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5301.586699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5306.442681] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5306.735881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5309.414191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.414711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5309.416653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5309.431249] Lustre: Skipped 1 previous similar message [ 5309.458142] Lustre: Skipped 1 previous similar message [ 5319.654303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5319.663506] Lustre: Skipped 3 previous similar messages [ 5323.231442] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5323.549889] Lustre: server umount lustre-OST0000 complete [ 5328.356436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5328.366657] LustreError: Skipped 1 previous similar message [ 5328.369828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.807136] Lustre: server umount lustre-MDT0000 complete [ 5335.013416] LustreError: 157089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.026982] LustreError: 157089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5337.669578] LustreError: 157073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385780 with bad export cookie 2558266382593711212 [ 5337.678614] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5337.682470] LustreError: 157073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5338.243437] Lustre: server umount lustre-MDT0001 complete [ 5345.825508] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:36:27 (1780385787) [ 5359.663733] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 5368.598695] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5399.233559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5399.416580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5399.439732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5399.486402] Lustre: lustre-MDT0000: new disk, initializing [ 5399.553343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.568805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5402.843738] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5411.357307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.470122] Lustre: 163398:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5411.691818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5415.708339] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5421.161867] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5423.805870] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5428.669699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5428.833184] Lustre: 165147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5433.575901] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5434.965076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5434.980188] Lustre: Skipped 1 previous similar message [ 5435.003128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5435.128560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5439.651521] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5445.604616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.606480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.618881] LustreError: Skipped 1 previous similar message [ 5445.625341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5445.630371] Lustre: Skipped 2 previous similar messages [ 5445.641117] Lustre: Skipped 4 previous similar messages [ 5451.491042] Lustre: server umount lustre-OST0000 complete [ 5455.840397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.852192] Lustre: Skipped 1 previous similar message [ 5460.677520] Lustre: server umount lustre-MDT0000 complete [ 5460.965955] LustreError: 163405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5460.982507] LustreError: 163405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5463.369121] LustreError: 163390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780385906 with bad export cookie 2558266382593712094 [ 5463.369969] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.378894] LustreError: 163390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5463.645613] Lustre: server umount lustre-MDT0001 complete [ 5469.910715] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:38:31 (1780385911) [ 5481.186257] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 5488.118808] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5513.351253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.530623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5513.549114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5513.556092] Lustre: Skipped 3 previous similar messages [ 5513.604605] Lustre: lustre-MDT0000: new disk, initializing [ 5513.606126] Lustre: Skipped 2 previous similar messages [ 5513.651951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5513.654944] Lustre: Skipped 2 previous similar messages [ 5513.668876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5516.639434] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5524.329602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.424522] Lustre: 169318:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5524.596775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5528.100570] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5533.548051] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5535.821930] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5540.223029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5540.473936] Lustre: 171066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5544.877556] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5549.697811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5551.199041] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5559.779234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5559.784186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.785490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5559.785496] Lustre: Skipped 4 previous similar messages [ 5559.812899] Lustre: Skipped 1 previous similar message [ 5562.320159] Lustre: server umount lustre-OST0000 complete [ 5565.408668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5571.531233] Lustre: server umount lustre-MDT0000 complete [ 5574.232859] LustreError: 169309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386017 with bad export cookie 2558266382593713172 [ 5574.234726] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5574.241356] LustreError: 169309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5574.517931] Lustre: server umount lustre-MDT0001 complete [ 5582.950273] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 5585.211819] Key type lgssc unregistered [ 5585.521493] LNet: 172792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5585.528518] LNetError: 172792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5585.542216] LNet: Removed LNI 192.168.201.149@tcp [ 5586.227609] Key type .llcrypt unregistered [ 5586.236084] Key type ._llcrypt unregistered [ 5602.831093] Key type ._llcrypt registered [ 5602.833175] Key type .llcrypt registered [ 5602.913433] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5603.715696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5603.733755] alg: No test for adler32 (adler32-zlib) [ 5604.752259] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5604.919586] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 5606.559192] Key type lgssc registered [ 5607.428572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5613.443849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5613.455537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.771592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.368684] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5624.292201] LustreError: 174170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5624.312293] LustreError: 174170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5625.140437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5625.368065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5628.745030] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5632.749853] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5635.594464] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5640.819691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5641.039775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5645.655126] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5650.033354] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5659.514063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5676.000190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5676.007795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.028040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5680.102645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5680.102660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.107379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5682.144791] Lustre: server umount lustre-OST0000 complete [ 5685.217356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5685.232820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5685.236963] Lustre: Skipped 2 previous similar messages [ 5690.336547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5690.345532] Lustre: Skipped 1 previous similar message [ 5691.256461] Lustre: server umount lustre-MDT0000 complete [ 5694.080319] LustreError: 174151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386137 with bad export cookie 7346335161401178988 [ 5694.088997] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5694.091993] LustreError: 174151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5694.322462] Lustre: server umount lustre-MDT0001 complete [ 5701.940438] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 5704.021862] Key type lgssc unregistered [ 5704.255905] LNet: 177365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5704.260408] LNetError: 177365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5704.275648] LNet: Removed LNI 192.168.201.149@tcp [ 5704.830436] Key type .llcrypt unregistered [ 5704.832445] Key type ._llcrypt unregistered [ 5721.982988] Key type ._llcrypt registered [ 5721.984618] Key type .llcrypt registered [ 5722.073901] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5722.793545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5722.936561] alg: No test for adler32 (adler32-zlib) [ 5723.900868] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 5724.061341] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 5725.727180] Key type lgssc registered [ 5726.669211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5732.547995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5732.568358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.889055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.734820] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5742.919243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.315399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.030380] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5751.432938] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5754.661863] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5759.861273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5760.132299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.164817] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5768.100303] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5779.634059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5789.153154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5789.158375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.170576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5790.188932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5790.196833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.199898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5790.211457] Lustre: Skipped 2 previous similar messages [ 5794.013455] Lustre: server umount lustre-OST0000 complete [ 5797.825800] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff957c099d6000.ost_server_uuid 50 [ 5798.944537] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5815.266229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5815.271342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5815.281377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5817.519038] Lustre: server umount lustre-MDT0000 complete [ 5820.452497] LustreError: 178724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386263 with bad export cookie 9278392621386584401 [ 5820.455891] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5820.463167] LustreError: 178724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5820.667311] Lustre: server umount lustre-MDT0001 complete [ 5826.959075] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:44:28 (1780386268) [ 5832.344510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5832.726732] LustreError: 182253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.745513] LustreError: 182253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5832.777553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.351677] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5837.793684] LustreError: 182254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5841.892567] LustreError: 182253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.563282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.837150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5846.233435] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5849.693917] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5851.943615] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5856.756571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.956962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5858.028697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5861.142916] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5865.636884] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5870.725023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5870.892812] Lustre: lustre-OST0001: new disk, initializing [ 5870.896889] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5870.903236] Lustre: 184750:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5875.694927] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5878.817372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5878.828483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5878.917375] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5881.531638] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5883.878619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5883.879184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.900698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.911970] Lustre: Skipped 2 previous similar messages [ 5883.918268] Lustre: Skipped 3 previous similar messages [ 5888.993528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5888.998278] Lustre: Skipped 2 previous similar messages [ 5889.579171] Lustre: server umount lustre-OST0000 complete [ 5892.094406] LustreError: 183849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.906106] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5894.081527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5896.694372] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5896.816897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5897.210903] LustreError: 185254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.221620] LustreError: 185254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5909.475414] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5909.478604] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5909.480407] LustreError: 183848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5909.480418] LustreError: 183848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5909.482023] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5909.527770] Lustre: Skipped 1 previous similar message [ 5914.984103] Lustre: server umount lustre-OST0001 complete [ 5919.716204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.721250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5919.733974] Lustre: Skipped 1 previous similar message [ 5924.027548] Lustre: server umount lustre-MDT0000 complete [ 5926.809406] LustreError: 182972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386370 with bad export cookie 9278392621386585472 [ 5926.814422] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5926.823843] LustreError: 182972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5927.039659] Lustre: server umount lustre-MDT0001 complete [ 5942.661411] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:46:24 (1780386384) [ 5946.521128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.696859] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5946.722907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5946.856195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5946.859419] Lustre: Skipped 1 previous similar message [ 5949.464462] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5954.952422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5955.010358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5955.025368] Lustre: 188038:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5957.980984] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5961.365344] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5963.549125] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5968.103362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.214424] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5971.744613] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5975.001575] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5977.679324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5979.107552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5979.195629] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5979.357380] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5979.362282] Lustre: Skipped 2 previous similar messages [ 5983.014685] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5986.948884] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5991.399061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.411847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5991.414259] Lustre: Skipped 1 previous similar message [ 5991.419351] Lustre: Skipped 5 previous similar messages [ 5994.710235] Lustre: server umount lustre-OST0000 complete [ 5996.518078] LustreError: 189662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5996.523668] LustreError: 189662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6011.880759] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6013.828102] Lustre: server umount lustre-OST0001 complete [ 6022.544129] Lustre: server umount lustre-MDT0000 complete [ 6024.777248] LustreError: 188033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386467 with bad export cookie 9278392621386586872 [ 6024.777248] LustreError: 188032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386467 with bad export cookie 9278392621386586872 [ 6024.781287] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6024.786789] LustreError: 188033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6024.991807] Lustre: server umount lustre-MDT0001 complete [ 6039.563724] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:48:00 (1780386480) [ 6049.981738] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 6056.582387] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6077.318984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.456810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6077.480489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6077.540104] Lustre: lustre-MDT0000: new disk, initializing [ 6077.605776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.620877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6080.081643] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6086.827696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6086.893811] Lustre: 195637:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6086.945866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6086.952252] Lustre: Skipped 1 previous similar message [ 6087.023698] Lustre: lustre-MDT0001: new disk, initializing [ 6087.105222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6087.109325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6089.163202] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6093.979759] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6095.928681] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6099.607976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6099.748959] Lustre: lustre-OST0000: new disk, initializing [ 6099.751853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6099.756784] Lustre: 197377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6102.860232] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6105.108737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6105.114688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6105.207848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6107.723546] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6109.777252] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6109.892897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6112.034323] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6112.198108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6115.300279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.302177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6115.307039] Lustre: Skipped 5 previous similar messages [ 6115.308614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6115.314986] Lustre: Skipped 9 previous similar messages [ 6115.333059] LustreError: Skipped 1 previous similar message [ 6119.625640] Lustre: server umount lustre-OST0000 complete [ 6130.303360] Lustre: server umount lustre-MDT0000 complete [ 6130.656216] LustreError: 195648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.665936] LustreError: 195648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6132.408905] LustreError: 195630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386575 with bad export cookie 9278392621386588433 [ 6132.410184] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6132.418060] LustreError: 195630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6132.615486] Lustre: server umount lustre-MDT0001 complete [ 6137.404721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.747838] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6144.369092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.795617] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6149.941765] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6152.187652] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6156.326159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6159.455800] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6162.151490] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6164.678463] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6164.836868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6167.369208] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6169.649374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 6177.003263] Lustre: server umount lustre-OST0000 complete [ 6179.817682] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6181.854151] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6206.681119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6206.838372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6206.841842] Lustre: Skipped 5 previous similar messages [ 6209.485441] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6211.737339] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6212.505649] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6213.992048] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.002395] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6219.233861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6219.236577] LustreError: Skipped 3 previous similar messages [ 6219.238257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6219.242130] Lustre: Skipped 4 previous similar messages [ 6219.244220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6219.246591] Lustre: Skipped 6 previous similar messages [ 6220.302806] Lustre: server umount lustre-OST0000 complete [ 6236.127227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6236.241358] Lustre: server umount lustre-MDT0000 complete [ 6237.949465] LustreError: 199427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386681 with bad export cookie 9278392621386589315 [ 6237.953040] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6237.953859] LustreError: 199427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6238.101105] Lustre: server umount lustre-MDT0001 complete [ 6241.805900] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:51:23 (1780386683) [ 6244.694666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6244.900446] LustreError: 204899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.908339] LustreError: 204899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6246.647360] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6250.528567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.313667] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6254.863155] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6256.711367] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6259.715399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6262.707535] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6265.152796] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6266.983386] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6271.213992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6272.895182] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6273.002879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6276.048510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6276.128960] Lustre: lustre-OST0001: new disk, initializing [ 6276.131070] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6276.135434] Lustre: 207920:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6278.735725] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6282.561658] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6283.793381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6283.797117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6283.836860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6284.304059] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6284.419392] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6286.573430] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6286.699313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6293.910589] Lustre: server umount lustre-OST0001 complete [ 6296.984301] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6297.107420] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6299.120586] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6299.212743] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6323.772684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6326.297082] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6328.606437] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6329.327496] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6329.338737] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6329.339243] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6329.373828] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6336.390614] Lustre: server umount lustre-OST0001 complete [ 6349.795460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.798425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6349.800521] Lustre: Skipped 7 previous similar messages [ 6349.814152] Lustre: Skipped 13 previous similar messages [ 6355.052158] Lustre: server umount lustre-OST0000 complete [ 6358.497983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6358.501226] LustreError: Skipped 4 previous similar messages [ 6371.295139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6371.384485] Lustre: server umount lustre-MDT0000 complete [ 6373.032222] LustreError: 204879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386816 with bad export cookie 9278392621386590253 [ 6373.036196] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.037185] LustreError: 204879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6379.291471] Lustre: server umount lustre-MDT0001 complete [ 6390.098557] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:53:51 (1780386831) [ 6393.370553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.540162] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6393.563964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6395.432699] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6399.387348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.437593] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6399.448576] Lustre: 212859:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6399.471042] LustreError: 212872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.478504] LustreError: 212872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 6401.108942] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6404.096421] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6406.120915] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6409.553599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6409.673574] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6412.329232] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6414.641786] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6424.142385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.200120] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6426.716520] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6429.104961] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6429.822432] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6431.541420] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6431.633132] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6433.308056] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6433.405296] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6434.901332] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff957c06fda800.ost_server_uuid 50 [ 6435.526887] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff957c06fda800.ost_server_uuid in FULL state after 0 sec [ 6435.963354] Lustre: Permanently deactivating lustre-OST0001 [ 6435.973791] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6436.945048] Lustre: Permanently reactivating lustre-OST0001 [ 6444.228628] Lustre: server umount lustre-OST0001 complete [ 6460.383078] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6460.448501] Lustre: server umount lustre-OST0000 complete [ 6468.272623] Lustre: server umount lustre-MDT0000 complete [ 6470.042336] LustreError: 212854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386913 with bad export cookie 9278392621386591541 [ 6470.046624] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6470.048444] LustreError: 212854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6470.232797] Lustre: server umount lustre-MDT0001 complete [ 6480.540342] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:55:22 (1780386922) [ 6484.116675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.290434] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6484.302030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6484.305316] Lustre: Skipped 5 previous similar messages [ 6484.394955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6484.398988] Lustre: Skipped 9 previous similar messages [ 6486.234108] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6490.010436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.068576] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6490.081973] Lustre: 218966:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6491.783691] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6494.258350] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6495.922810] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6499.061772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6499.165968] Lustre: Permanently deactivating lustre-OST0000 [ 6499.172216] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6501.636751] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6501.656918] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6501.666480] LustreError: 220949:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 6501.667792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6503.963846] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6506.806515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6509.165103] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6510.251189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6511.410083] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6512.242783] Lustre: Permanently reactivating lustre-OST0000 [ 6518.783799] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6518.787753] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.792127] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6518.794481] Lustre: Skipped 1 previous similar message [ 6518.809273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.815338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6527.616207] Lustre: server umount lustre-OST0001 complete [ 6535.320300] Lustre: server umount lustre-OST0000 complete [ 6543.087264] Lustre: server umount lustre-MDT0000 complete [ 6544.457628] LustreError: 219688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780386987 with bad export cookie 9278392621386593193 [ 6544.462736] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.466364] LustreError: 219688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6544.599397] Lustre: server umount lustre-MDT0001 complete [ 6548.544973] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6549.722160] Key type lgssc unregistered [ 6549.871540] LNet: 223348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6549.874265] LNetError: 223348:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6549.881332] LNet: Removed LNI 192.168.201.149@tcp [ 6550.222416] Key type .llcrypt unregistered [ 6550.224061] Key type ._llcrypt unregistered [ 6560.144108] Key type ._llcrypt registered [ 6560.145592] Key type .llcrypt registered [ 6560.193314] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6560.542572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6560.550739] alg: No test for adler32 (adler32-zlib) [ 6561.426448] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6561.523318] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 6563.111160] Key type lgssc registered [ 6563.517095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6567.055719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6567.064722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6568.267822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.726714] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6572.837292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6572.880766] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6572.893261] Lustre: 224767:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6572.897387] Lustre: Permanently deactivating lustre-MDT0001 [ 6572.902647] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6573.020240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6573.206260] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6573.209832] LustreError: 224808:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6573.214851] LustreError: 224808:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6573.218370] LustreError: 224808:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 6574.510293] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6576.540185] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6578.004460] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6580.583028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6580.715280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.009421] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6583.778884] LustreError: 226389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.784383] LustreError: 226389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6583.785179] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6585.423629] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6588.311359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.410671] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6590.616066] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6592.971844] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6593.515648] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6593.519301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6593.521568] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6599.158405] Lustre: Permanently reactivating lustre-MDT0001 [ 6599.170613] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6599.173299] Lustre: Skipped 2 previous similar messages [ 6602.426954] Lustre: lustre-MDT0001: Client b7c00367-00d2-4234-97a2-9c9ab638da0f (at 192.168.201.49@tcp) reconnecting [ 6604.949399] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6605.692470] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6607.878605] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6607.984443] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6609.695343] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6609.792394] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6610.964500] Lustre: Permanently deactivating lustre-MDT0001 [ 6610.971392] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6610.974063] Lustre: Skipped 2 previous similar messages [ 6618.155342] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6623.147593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.148449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6623.154043] Lustre: Skipped 3 previous similar messages [ 6623.157112] Lustre: Skipped 2 previous similar messages [ 6627.426259] Lustre: server umount lustre-MDT0000 complete [ 6628.322667] LustreError: 225504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6628.327524] LustreError: 225504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6629.261336] LustreError: 228673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387072 with bad export cookie 18305376208808274207 [ 6629.265045] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6629.271173] LustreError: 228673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6629.454519] Lustre: server umount lustre-MDT0001 complete [ 6637.369970] Lustre: server umount lustre-OST0000 complete [ 6641.680558] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:58:03 (1780387083) [ 6645.548652] Lustre: server umount lustre-OST0001 complete [ 6647.825734] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 6651.209412] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6665.024227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.126902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6665.130404] Lustre: Skipped 2 previous similar messages [ 6665.141934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6665.181844] Lustre: lustre-MDT0000: new disk, initializing [ 6665.219817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.229548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6666.802482] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6671.622883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6671.680401] Lustre: 232408:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6671.706259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6671.708823] Lustre: Skipped 1 previous similar message [ 6671.748697] Lustre: lustre-MDT0001: new disk, initializing [ 6671.819752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6671.825942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6673.420546] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6677.169484] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6678.782202] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6681.658077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6681.821421] Lustre: lustre-OST0000: new disk, initializing [ 6681.823894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6681.827191] Lustre: 234149:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6681.866409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.871664] Lustre: Skipped 1 previous similar message [ 6684.326801] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6688.124608] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6690.320725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6690.326394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6690.370474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6694.477118] LustreError: 234689:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6699.183675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6699.272652] Lustre: lustre-OST0001: new disk, initializing [ 6699.275303] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6699.279343] Lustre: 235148:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6699.345058] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6699.557858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6699.560658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6699.590681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6701.728224] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6704.575135] LustreError: 234689:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6705.106107] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6708.722756] LustreError: 232414:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6708.747475] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6718.815060] LustreError: 232414:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6719.970817] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6719.973884] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6719.975609] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6719.977993] Lustre: Skipped 1 previous similar message [ 6719.980134] Lustre: Skipped 1 previous similar message [ 6723.066081] Lustre: lustre-OST0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 6725.091635] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6725.094587] Lustre: Skipped 1 previous similar message [ 6725.886282] Lustre: server umount lustre-OST0001 complete [ 6730.211260] LustreError: 234132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6730.218392] LustreError: 234132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6735.332054] LustreError: 234134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.340761] LustreError: 234134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6740.451159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.452366] LustreError: 234134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.457016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6740.461759] LustreError: 234134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6740.464211] Lustre: Skipped 1 previous similar message [ 6750.691508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.692200] LustreError: 234134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.694577] Lustre: Skipped 3 previous similar messages [ 6750.701336] LustreError: 234134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6752.223123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6752.282186] Lustre: server umount lustre-OST0000 complete [ 6753.760033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6753.762805] LustreError: Skipped 1 previous similar message [ 6753.764417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6753.768406] Lustre: Skipped 1 previous similar message [ 6759.910481] Lustre: server umount lustre-MDT0000 complete [ 6761.386900] LustreError: 234148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387204 with bad export cookie 18305376208808277014 [ 6761.390676] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6761.392151] LustreError: 234148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6767.545519] Lustre: server umount lustre-MDT0001 complete [ 6771.267714] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6772.444162] Key type lgssc unregistered [ 6772.609274] LNet: 237070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6772.612739] LNetError: 237070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6772.624968] LNet: Removed LNI 192.168.201.149@tcp [ 6772.957580] Key type .llcrypt unregistered [ 6772.959159] Key type ._llcrypt unregistered [ 6788.460442] Key type ._llcrypt registered [ 6788.462164] Key type .llcrypt registered [ 6788.506222] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6788.979244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6788.999092] alg: No test for adler32 (adler32-zlib) [ 6789.868884] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6789.967161] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 6791.559114] Key type lgssc registered [ 6791.941977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6794.626717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6794.632550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.736369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6795.744805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6795.807630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6797.202376] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6800.411303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.462180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6800.473701] Lustre: 239233:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6800.577838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6802.088954] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6804.114503] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6805.486142] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6808.128214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6808.217081] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6808.308291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.394700] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6812.441418] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6815.779261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6861.792682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6861.795191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.801251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6861.803479] Lustre: Skipped 1 previous similar message [ 6863.718510] Lustre: server umount lustre-OST0000 complete [ 6865.578453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6870.546117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.697162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.897257] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6875.087438] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6881.956977] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6881.962522] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6881.962626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6912.993202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6912.994119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6912.996958] LustreError: Skipped 1 previous similar message [ 6912.997574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6913.002482] Lustre: Skipped 2 previous similar messages [ 6914.851732] Lustre: server umount lustre-OST0000 complete [ 6918.112867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.114937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6918.118119] Lustre: Skipped 1 previous similar message [ 6918.123223] Lustre: Skipped 2 previous similar messages [ 6922.588129] Lustre: server umount lustre-MDT0000 complete [ 6924.097394] LustreError: 239226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387367 with bad export cookie 16548614598561794176 [ 6924.100434] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.101401] LustreError: 239226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6924.220280] Lustre: server umount lustre-MDT0001 complete [ 6928.242818] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6929.457989] Key type lgssc unregistered [ 6929.599416] LNet: 244190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6929.603657] LNetError: 244190:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6929.612313] LNet: Removed LNI 192.168.201.149@tcp [ 6929.957530] Key type .llcrypt unregistered [ 6929.958901] Key type ._llcrypt unregistered [ 6940.499929] Key type ._llcrypt registered [ 6940.501948] Key type .llcrypt registered [ 6940.555939] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6940.922573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6940.930956] alg: No test for adler32 (adler32-zlib) [ 6941.808939] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 6941.911100] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 6943.519257] Key type lgssc registered [ 6943.977358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6947.045847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6947.054443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.240366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6949.842221] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6953.195959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.358591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6954.883403] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6956.957696] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6958.398312] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6961.090604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6961.244738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6963.249967] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6965.277298] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6973.545414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6989.284377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6989.286150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.286882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6989.291221] LustreError: Skipped 1 previous similar message [ 6989.295679] Lustre: Skipped 1 previous similar message [ 6990.868622] Lustre: server umount lustre-OST0000 complete [ 6994.400695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.402736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6994.406904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.410359] Lustre: Skipped 2 previous similar messages [ 6998.514063] Lustre: server umount lustre-MDT0000 complete [ 6999.520475] LustreError: 245569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.528090] LustreError: 245569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7000.023216] LustreError: 245550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387443 with bad export cookie 8461866335302115270 [ 7000.025925] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7000.027769] LustreError: 245550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7000.142804] Lustre: server umount lustre-MDT0001 complete [ 7004.364489] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7005.627584] Key type lgssc unregistered [ 7005.763557] LNet: 249534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7005.768264] LNetError: 249534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7005.778368] LNet: Removed LNI 192.168.201.149@tcp [ 7006.104807] Key type .llcrypt unregistered [ 7006.106315] Key type ._llcrypt unregistered [ 7013.963967] Key type ._llcrypt registered [ 7013.965414] Key type .llcrypt registered [ 7014.009619] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7014.430503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7014.437784] alg: No test for adler32 (adler32-zlib) [ 7015.303185] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7015.400088] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7016.983165] Key type lgssc registered [ 7017.397891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7020.598683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7020.607262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7021.829978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7023.727641] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7027.260153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.453443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7029.005518] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7031.091264] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7032.486152] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7035.024364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7035.188053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.554307] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7039.771856] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7047.462250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7057.891024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7057.892326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.896291] LustreError: Skipped 1 previous similar message [ 7057.901945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7057.905230] Lustre: Skipped 1 previous similar message [ 7059.994668] Lustre: server umount lustre-OST0000 complete [ 7063.009269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.010692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.014414] Lustre: Skipped 2 previous similar messages [ 7063.018612] Lustre: Skipped 1 previous similar message [ 7067.481078] Lustre: server umount lustre-MDT0000 complete [ 7068.128375] LustreError: 250717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.136657] LustreError: 250717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7068.835325] LustreError: 250697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387512 with bad export cookie 1870500078513632715 [ 7068.838520] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7068.839354] LustreError: 250697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7068.946457] Lustre: server umount lustre-MDT0001 complete [ 7072.625787] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7073.712403] Key type lgssc unregistered [ 7073.838442] LNet: 254054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7073.841030] LNetError: 254054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7073.850642] LNet: Removed LNI 192.168.201.149@tcp [ 7074.146065] Key type .llcrypt unregistered [ 7074.147205] Key type ._llcrypt unregistered [ 7083.190285] Key type ._llcrypt registered [ 7083.191530] Key type .llcrypt registered [ 7083.234894] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7083.706729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7083.717909] alg: No test for adler32 (adler32-zlib) [ 7084.575398] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7084.661260] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7086.239180] Key type lgssc registered [ 7086.680955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7089.840924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7089.848311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.033535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.401398] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7095.277952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.429712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7096.767300] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7098.517460] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7099.686024] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7101.957278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7102.062132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7103.896668] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7105.638941] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7117.414481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7129.056890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7129.059038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7129.064094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7131.104080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7131.106643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7131.110498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7133.152298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7133.154621] Lustre: Skipped 1 previous similar message [ 7134.936955] Lustre: server umount lustre-OST0000 complete [ 7136.736162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7136.739289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.745389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7142.356522] Lustre: server umount lustre-MDT0000 complete [ 7143.392917] LustreError: 255431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7143.400263] LustreError: 255431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7143.650111] LustreError: 255410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387586 with bad export cookie 13585349280007842210 [ 7143.652224] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7143.653409] LustreError: 255410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7143.762106] Lustre: server umount lustre-MDT0001 complete [ 7147.342700] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7148.398258] Key type lgssc unregistered [ 7148.524526] LNet: 259394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7148.528214] LNetError: 259394:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7148.535286] LNet: Removed LNI 192.168.201.149@tcp [ 7148.828118] Key type .llcrypt unregistered [ 7148.829163] Key type ._llcrypt unregistered [ 7155.822550] Key type ._llcrypt registered [ 7155.823811] Key type .llcrypt registered [ 7155.858395] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7156.203677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7156.253025] alg: No test for adler32 (adler32-zlib) [ 7157.107651] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7157.180039] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7158.767095] Key type lgssc registered [ 7159.129318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7161.700352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7161.706051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.875039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.198661] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7167.011822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.142147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7168.515582] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7170.451781] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7171.687412] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7174.007526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7174.136610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7176.136818] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7177.958640] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7188.455236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7192.543803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7192.546500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7192.551254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7193.570119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7193.570301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.570408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7194.261457] Lustre: server umount lustre-OST0000 complete [ 7197.650105] Lustre: server umount lustre-MDT0000 complete [ 7198.175880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7198.180025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.186116] LustreError: 260593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.191530] LustreError: 260593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7198.689247] LustreError: 260579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.944295] LustreError: 261331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387642 with bad export cookie 16598747704657796150 [ 7198.947995] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7198.948275] LustreError: 261331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7199.063663] Lustre: server umount lustre-MDT0001 complete [ 7202.704222] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7203.760731] Key type lgssc unregistered [ 7203.891332] LNet: 263951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7203.893564] LNetError: 263951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7203.900333] LNet: Removed LNI 192.168.201.149@tcp [ 7204.211599] Key type .llcrypt unregistered [ 7204.212934] Key type ._llcrypt unregistered [ 7207.854310] Key type ._llcrypt registered [ 7207.855572] Key type .llcrypt registered [ 7207.904647] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7213.414060] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 7219.478392] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7219.906286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7219.926440] alg: No test for adler32 (adler32-zlib) [ 7220.776775] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7220.857680] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7222.439118] Key type lgssc registered [ 7222.823946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7234.876059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7234.889844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.029595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7236.043343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7236.082778] Lustre: lustre-MDT0000: new disk, initializing [ 7236.111543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7236.118685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7237.585562] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7242.072671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.117340] Lustre: 267425:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7242.142608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7242.145042] Lustre: Skipped 1 previous similar message [ 7242.182485] Lustre: lustre-MDT0001: new disk, initializing [ 7242.220961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7242.232623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7242.236842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7243.697991] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7246.945792] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7248.213513] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7250.662897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.764884] Lustre: lustre-OST0000: new disk, initializing [ 7250.766965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7250.769957] Lustre: 269166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7250.809520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7252.867055] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7253.119740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7253.123886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7253.162962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7256.339968] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7257.801746] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7257.889352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7259.350094] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7259.423928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7263.714071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7263.714515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7263.716957] LustreError: Skipped 1 previous similar message [ 7263.721055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7263.723751] Lustre: Skipped 1 previous similar message [ 7266.257269] Lustre: server umount lustre-OST0000 complete [ 7267.807806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7267.811648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.816572] Lustre: Skipped 1 previous similar message [ 7267.818883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7268.832379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7268.833116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.629919] Lustre: server umount lustre-MDT0000 complete [ 7274.909163] LustreError: 267417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387718 with bad export cookie 8584874144861242903 [ 7274.912467] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7274.913560] LustreError: 267417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7275.025809] Lustre: server umount lustre-MDT0001 complete [ 7277.922994] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:08:39 (1780387719) [ 7280.319500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.479564] LustreError: 271427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.486138] LustreError: 271427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7280.510056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7281.889113] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7285.029881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.187066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7286.521360] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7288.321804] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7289.630664] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7292.114277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7293.976445] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7295.648166] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7310.818482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7310.819625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.820411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7310.820415] Lustre: Skipped 1 previous similar message [ 7310.830801] Lustre: Skipped 1 previous similar message [ 7315.211354] Lustre: server umount lustre-OST0000 complete [ 7320.032985] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7320.036521] LustreError: Skipped 1 previous similar message [ 7320.038434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.043387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7320.045078] Lustre: Skipped 1 previous similar message [ 7322.709470] Lustre: server umount lustre-MDT0000 complete [ 7323.973948] LustreError: 271408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387767 with bad export cookie 8584874144861243771 [ 7323.975981] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7323.976709] LustreError: 272148:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 7323.978404] LustreError: 271408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7324.092102] Lustre: server umount lustre-MDT0001 complete [ 7327.520091] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7328.469308] Key type lgssc unregistered [ 7328.587379] LNet: 274626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7328.590155] LNetError: 274626:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7328.598214] LNet: Removed LNI 192.168.201.149@tcp [ 7328.885633] Key type .llcrypt unregistered [ 7328.886489] Key type ._llcrypt unregistered [ 7338.061800] Key type ._llcrypt registered [ 7338.063157] Key type .llcrypt registered [ 7338.104949] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 7344.528266] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7344.974826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7344.994331] alg: No test for adler32 (adler32-zlib) [ 7345.846154] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7345.939337] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7347.527164] Key type lgssc registered [ 7347.914643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7364.331524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7364.343551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.441177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7365.455776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7365.504920] Lustre: lustre-MDT0000: new disk, initializing [ 7365.547684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7365.556345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7367.027715] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7371.364384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.413062] Lustre: 278389:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7371.443473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7371.446053] Lustre: Skipped 1 previous similar message [ 7371.489480] Lustre: lustre-MDT0001: new disk, initializing [ 7371.530656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7371.552663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7371.557858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7372.936740] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7376.178820] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7377.486354] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7379.760218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7379.865450] Lustre: lustre-OST2710: new disk, initializing [ 7379.867416] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7379.869692] Lustre: 280130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7379.900055] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7381.887341] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7383.546117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7383.549762] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7383.589173] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7385.119158] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7387.481431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7387.546624] Lustre: lustre-OST03e8: new disk, initializing [ 7387.548626] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7387.552289] Lustre: 281079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7387.585299] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7389.512292] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7389.634390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7389.638735] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7389.674802] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7391.326862] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7394.102995] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7394.183557] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7395.537465] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7395.615793] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7396.942171] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7397.021101] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7398.400906] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7398.486486] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7399.904600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7399.907456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7399.908502] Lustre: Skipped 3 previous similar messages [ 7399.909915] Lustre: Skipped 2 previous similar messages [ 7405.027032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7405.030554] Lustre: Skipped 4 previous similar messages [ 7405.674514] Lustre: server umount lustre-MDT0000 complete [ 7407.027183] LustreError: 280129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387850 with bad export cookie 12303211443503996405 [ 7407.030474] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7407.030811] LustreError: 280129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7407.164038] Lustre: server umount lustre-MDT0001 complete [ 7418.377653] Lustre: server umount lustre-OST2710 complete [ 7428.955943] Lustre: server umount lustre-OST03e8 complete [ 7434.603138] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 7437.686582] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7449.366719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.455630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7449.465983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7449.504400] Lustre: lustre-MDT0000: new disk, initializing [ 7449.527529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7449.533923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7450.787874] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7454.765458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.806127] Lustre: 286103:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7454.899413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7456.132568] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7459.160755] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7460.298912] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7462.548731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7462.650046] Lustre: lustre-OST0000: new disk, initializing [ 7462.651798] Lustre: Skipped 1 previous similar message [ 7462.654084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7462.656374] Lustre: Skipped 2 previous similar messages [ 7462.659233] Lustre: 287846:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7462.696706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7462.699790] Lustre: Skipped 1 previous similar message [ 7463.927981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7463.930642] Lustre: Skipped 1 previous similar message [ 7463.932718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7463.967080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7464.491061] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7467.475318] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7468.779389] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7468.854659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7470.145878] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7470.217166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7474.145700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7474.145840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7474.148342] LustreError: Skipped 1 previous similar message [ 7474.154562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7474.157430] Lustre: Skipped 1 previous similar message [ 7476.945331] Lustre: server umount lustre-OST0000 complete [ 7479.264606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.265164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7479.269460] Lustre: Skipped 2 previous similar messages [ 7484.247669] Lustre: server umount lustre-MDT0000 complete [ 7485.426256] LustreError: 286097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387928 with bad export cookie 12303211443503998085 [ 7485.428822] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7485.430215] LustreError: 286097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7485.545827] Lustre: server umount lustre-MDT0001 complete [ 7488.313298] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:12:10 (1780387930) [ 7488.829322] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7489.385040] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:12:11 (1780387931) [ 7490.325847] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7499.423960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.530909] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7499.541043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7499.600558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7500.886890] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7503.837859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7503.866558] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7503.875826] Lustre: 291534:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7505.275291] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7521.249993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7521.310276] Lustre: Denying initial registration attempt from nid 192.168.201.149@tcp, specified as failover [ 7521.314139] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7521.317542] LustreError: 292953:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 7521.319793] LustreError: 292953:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7521.322180] LustreError: 292953:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7521.347582] Lustre: server umount lustre-OST0000 complete [ 7521.349539] LustreError: 292953:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7524.319753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7524.323243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.329282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7524.331264] Lustre: Skipped 1 previous similar message [ 7529.614962] Lustre: server umount lustre-MDT0000 complete [ 7530.713084] LustreError: 291526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780387973 with bad export cookie 12303211443503998953 [ 7530.715409] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7530.716369] LustreError: 291526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7530.819955] Lustre: server umount lustre-MDT0001 complete [ 7534.116503] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7535.088107] Key type lgssc unregistered [ 7535.221337] LNet: 294055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7535.224099] LNetError: 294055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7535.233283] LNet: Removed LNI 192.168.201.149@tcp [ 7535.512401] Key type .llcrypt unregistered [ 7535.513377] Key type ._llcrypt unregistered [ 7539.078631] Key type ._llcrypt registered [ 7539.080255] Key type .llcrypt registered [ 7539.124807] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7539.479442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7539.534162] alg: No test for adler32 (adler32-zlib) [ 7540.389125] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7540.463716] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7542.039177] Key type lgssc registered [ 7542.388172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7566.086616] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7569.020481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7569.026862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.115600] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7570.123783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7570.189704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.413262] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7574.211198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.246460] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7574.259262] Lustre: 296759:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7574.351279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7575.557495] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7577.236742] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7578.354571] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7593.505736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7593.563748] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7593.620075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.242398] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7596.777907] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7597.410266] Lustre: server umount lustre-OST0000 complete [ 7599.072613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7599.073251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7599.075693] Lustre: Skipped 1 previous similar message [ 7599.079829] Lustre: Skipped 1 previous similar message [ 7602.668629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7602.670531] Lustre: Skipped 1 previous similar message [ 7604.621443] Lustre: server umount lustre-MDT0000 complete [ 7605.740261] LustreError: 296750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388048 with bad export cookie 12950598574750816787 [ 7605.743046] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7605.744031] LustreError: 296750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7605.849468] Lustre: server umount lustre-MDT0001 complete [ 7609.072543] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7610.055362] Key type lgssc unregistered [ 7610.183349] LNet: 299963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7610.186393] LNetError: 299963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7610.193549] LNet: Removed LNI 192.168.201.149@tcp [ 7610.482893] Key type .llcrypt unregistered [ 7610.483970] Key type ._llcrypt unregistered [ 7618.995556] Key type ._llcrypt registered [ 7618.996910] Key type .llcrypt registered [ 7619.036370] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7619.340814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7619.387632] alg: No test for adler32 (adler32-zlib) [ 7620.239295] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7620.315294] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7621.895119] Key type lgssc registered [ 7622.253345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7624.821061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7624.828799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7625.987951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7627.273382] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7630.103131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.213772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7631.462244] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7633.182905] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7634.335802] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7638.562029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7638.671696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7640.512351] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7642.293146] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7655.904231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7655.906934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.911743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7656.418446] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.418677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7656.425136] Lustre: Skipped 2 previous similar messages [ 7657.978095] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 7660.893126] Lustre: server umount lustre-MDT0000 complete [ 7661.536709] LustreError: 301336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.542390] LustreError: 301336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7662.262600] LustreError: 301322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388105 with bad export cookie 1276555586505183999 [ 7662.264707] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.266621] LustreError: 301322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7663.093228] LustreError: 301337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7663.101228] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 7666.655811] LustreError: 301335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.656019] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.663967] Lustre: Skipped 1 previous similar message [ 7667.680045] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7667.682553] Lustre: Skipped 1 previous similar message [ 7668.457433] Lustre: server umount lustre-MDT0001 complete [ 7670.537169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7673.181612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.159717] LustreError: 300652:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006b9d2a4d x1866872193572480/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7688.275416] LustreError: 302943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.281683] LustreError: 302943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7688.304907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.326342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7689.720128] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7692.780469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7692.785134] LustreError: 304610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.907182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7693.925572] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7694.358858] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7696.200857] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7698.018911] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7699.827073] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7700.952429] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7701.471862] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7703.008065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7703.010246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.013395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7703.014813] Lustre: Skipped 1 previous similar message [ 7716.831150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7716.898962] Lustre: server umount lustre-OST0000 complete [ 7718.367815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7718.370871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.375190] Lustre: Skipped 1 previous similar message [ 7723.999960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7724.002457] Lustre: Skipped 8 previous similar messages [ 7724.243718] Lustre: server umount lustre-MDT0000 complete [ 7725.456261] LustreError: 301320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388168 with bad export cookie 1276555586505191160 [ 7725.459266] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7725.460630] LustreError: 301320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7725.569831] Lustre: server umount lustre-MDT0001 complete [ 7729.159112] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7730.201526] Key type lgssc unregistered [ 7730.329367] LNet: 307040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7730.332227] LNetError: 307040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7730.341284] LNet: Removed LNI 192.168.201.149@tcp [ 7730.640657] Key type .llcrypt unregistered [ 7730.641667] Key type ._llcrypt unregistered [ 7739.279087] Key type ._llcrypt registered [ 7739.280337] Key type .llcrypt registered [ 7739.313140] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7739.615740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7739.659329] alg: No test for adler32 (adler32-zlib) [ 7740.509890] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7740.592394] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7742.175160] Key type lgssc registered [ 7742.566736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7745.367969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7745.374607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.526166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7747.862744] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7750.792722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.906177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7752.068210] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7753.653463] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7754.756099] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7755.644596] LustreError: 308453:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7755.646493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.647487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.649075] LustreError: 308453:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7755.651824] Lustre: Skipped 1 previous similar message [ 7755.653786] LustreError: 308453:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 7760.800245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7760.802150] Lustre: Skipped 2 previous similar messages [ 7761.810863] Lustre: server umount lustre-MDT0000 complete [ 7762.966578] LustreError: 308404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388206 with bad export cookie 6106619108845950634 [ 7762.969338] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.973798] LustreError: 308404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7763.080429] Lustre: server umount lustre-MDT0001 complete [ 7767.537799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.624572] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7767.633735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7767.699057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.891957] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7771.564144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.596201] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7771.607458] Lustre: 310784:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7771.699412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7772.974135] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7774.739415] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7775.884669] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7780.257808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7780.323121] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7780.325702] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7780.328163] LustreError: 312437:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 7780.330099] LustreError: 312437:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7780.331867] LustreError: 312437:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7780.355986] Lustre: server umount lustre-OST0000 complete [ 7780.358034] LustreError: 312437:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7785.057843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7785.120526] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7785.179844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7785.990062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7786.945610] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7788.649041] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7791.095663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7791.123916] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7791.157041] Lustre: lustre-OST0001: new disk, initializing [ 7791.158418] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7791.160776] Lustre: 313864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7792.382562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7792.386864] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7792.420597] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7793.032094] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7794.722753] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7797.728361] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7797.730948] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7797.731356] LustreError: Skipped 1 previous similar message [ 7797.734547] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7797.737310] Lustre: Skipped 1 previous similar message [ 7801.661928] Lustre: server umount lustre-OST0001 complete [ 7802.849820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.849977] LustreError: 313061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.853345] Lustre: Skipped 1 previous similar message [ 7802.859087] LustreError: 313061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7802.861236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7802.864784] Lustre: Skipped 1 previous similar message [ 7807.969620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7807.970453] LustreError: 314553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7807.971499] Lustre: Skipped 1 previous similar message [ 7807.974907] LustreError: 314553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7813.091532] LustreError: 312903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.096440] LustreError: 312903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7817.183090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7817.231243] Lustre: server umount lustre-OST0000 complete [ 7822.815802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7822.818507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.823112] Lustre: Skipped 1 previous similar message [ 7822.824605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7822.826067] Lustre: Skipped 2 previous similar messages [ 7824.542973] Lustre: server umount lustre-MDT0000 complete [ 7825.735944] LustreError: 310776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388268 with bad export cookie 6106619108845951166 [ 7825.739299] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.739304] LustreError: 310776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7825.851111] Lustre: server umount lustre-MDT0001 complete [ 7829.191784] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7830.219188] Key type lgssc unregistered [ 7830.351330] LNet: 315690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7830.353541] LNetError: 315690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7830.364294] LNet: Removed LNI 192.168.201.149@tcp [ 7830.660141] Key type .llcrypt unregistered [ 7830.661699] Key type ._llcrypt unregistered [ 7860.548671] Key type ._llcrypt registered [ 7860.549685] Key type .llcrypt registered [ 7860.585793] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 7866.567793] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7866.902131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7866.964260] alg: No test for adler32 (adler32-zlib) [ 7867.817459] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 7867.909723] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7869.495147] Key type lgssc registered [ 7869.846457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7881.178870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7881.183506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.277073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7882.289292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7882.330469] Lustre: lustre-MDT0000: new disk, initializing [ 7882.354577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7882.361951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7883.699168] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7887.813366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7887.860055] Lustre: 320558:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7887.883690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7887.885618] Lustre: Skipped 1 previous similar message [ 7887.923887] Lustre: lustre-MDT0001: new disk, initializing [ 7887.955186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7887.966797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7887.970341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7889.308451] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7892.408233] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7893.593515] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7895.795717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7895.893176] Lustre: lustre-OST0000: new disk, initializing [ 7895.895105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7895.898446] Lustre: 322295:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7895.926154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7897.842765] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7900.661159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7900.664311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7900.691288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7900.838088] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7902.188547] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7902.256740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7903.547189] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7903.617541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7905.761522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7905.761608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7905.764656] LustreError: Skipped 1 previous similar message [ 7905.768836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7905.771181] Lustre: Skipped 1 previous similar message [ 7906.254842] Lustre: server umount lustre-OST0000 complete [ 7908.319871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7908.322068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7908.325710] Lustre: Skipped 1 previous similar message [ 7908.327626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7910.880064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7910.880488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.554575] Lustre: server umount lustre-MDT0000 complete [ 7914.693146] LustreError: 320551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388357 with bad export cookie 8156084552812098090 [ 7914.695936] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7914.696547] LustreError: 320551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7914.797499] Lustre: server umount lustre-MDT0001 complete [ 7917.227492] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:19:19 (1780388359) [ 7919.333956] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:19:21 (1780388361) [ 7921.936921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7922.086083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.275101] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7926.015553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.143353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7927.292555] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7928.953959] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7929.997661] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7932.308393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7932.386088] LustreError: 326491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7932.392152] LustreError: 326491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7934.268753] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7935.999265] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7945.784720] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7946.366616] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7946.902055] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7947.443337] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7948.104308] Lustre: Failing over lustre-MDT0000 [ 7948.335375] Lustre: server umount lustre-MDT0000 complete [ 7951.841798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.842156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7951.843123] LustreError: 324893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7951.846593] Lustre: Skipped 1 previous similar message [ 7952.889655] LustreError: 324895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7952.895432] LustreError: 324895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7956.960867] LustreError: 324898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7956.967829] LustreError: 324898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7961.253850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7961.295841] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7961.398914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.400846] Lustre: Skipped 1 previous similar message [ 7961.432350] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7962.724961] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7963.121317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7966.695360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7966.704625] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7966.723767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7967.879943] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7968.448345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7970.320612] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7971.006547] Lustre: Failing over lustre-MDT0000 [ 7971.153757] Lustre: server umount lustre-MDT0000 complete [ 7971.808978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7971.809828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7971.817046] Lustre: Skipped 1 previous similar message [ 7971.820856] LustreError: 325607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.827585] LustreError: 325607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7982.048265] LustreError: 325607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7982.054269] LustreError: 325607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7984.000313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.042520] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7984.136863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7984.165237] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7985.425601] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7988.721546] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7989.221715] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7989.224080] Lustre: Skipped 2 previous similar messages [ 7989.231846] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7989.251199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7990.460397] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7990.983241] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7994.338345] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.338775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7994.339307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7994.339313] Lustre: Skipped 1 previous similar message [ 7994.342018] Lustre: Skipped 4 previous similar messages [ 7999.318173] Lustre: server umount lustre-MDT0000 complete [ 7999.456065] LustreError: 328121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7999.462077] LustreError: 328121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8000.609201] LustreError: 324881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388443 with bad export cookie 8156084552812100778 [ 8000.612926] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8000.614908] LustreError: 324881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8000.728498] Lustre: server umount lustre-MDT0001 complete [ 8012.311315] Lustre: server umount lustre-OST0000 complete [ 8017.702600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.882270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.195690] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8021.952293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8023.279684] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8024.960989] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8026.069542] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8028.091895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.894021] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8031.442920] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8032.298276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8042.467835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.468037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8042.473052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8042.474512] LustreError: Skipped 1 previous similar message [ 8042.476561] Lustre: Skipped 3 previous similar messages [ 8052.191099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8052.249492] Lustre: server umount lustre-OST0000 complete [ 8057.825785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8057.827488] Lustre: Skipped 3 previous similar messages [ 8059.605157] Lustre: server umount lustre-MDT0000 complete [ 8060.827241] LustreError: 330642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388504 with bad export cookie 8156084552812101499 [ 8060.830263] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.833537] LustreError: 330642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8060.943267] Lustre: server umount lustre-MDT0001 complete [ 8064.672245] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8065.748352] Key type lgssc unregistered [ 8065.884335] LNet: 333834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8065.886491] LNetError: 333834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8065.899332] LNet: Removed LNI 192.168.201.149@tcp [ 8066.214064] Key type .llcrypt unregistered [ 8066.215925] Key type ._llcrypt unregistered [ 8075.497481] Key type ._llcrypt registered [ 8075.498314] Key type .llcrypt registered [ 8075.532837] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8075.837025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8075.869101] alg: No test for adler32 (adler32-zlib) [ 8076.709063] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8076.782896] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8078.359155] Key type lgssc registered [ 8078.678338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8080.998015] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8081.002791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.141621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8083.273726] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8085.820618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.935436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8087.090659] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8088.690035] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8089.738967] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8091.757618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.856167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.901480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8093.489509] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8095.037912] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8101.345387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.346103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8101.347918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.351458] Lustre: Skipped 2 previous similar messages [ 8105.754130] Lustre: server umount lustre-MDT0000 complete [ 8106.464963] LustreError: 336766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.471173] LustreError: 336766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8106.855190] LustreError: 335291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388550 with bad export cookie 8663211983964662452 [ 8106.858706] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8106.859430] LustreError: 335291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8106.958135] Lustre: server umount lustre-MDT0001 complete [ 8118.365489] Lustre: server umount lustre-OST0000 complete [ 8122.994180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8123.153996] LustreError: 338677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8123.178697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8124.503370] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8127.224906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.343742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8128.651854] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8130.426859] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8131.663260] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8133.850660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8135.860744] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8137.666232] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8148.327091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8163.295855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8163.298487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8163.304239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8163.306277] Lustre: Skipped 2 previous similar messages [ 8166.037668] Lustre: server umount lustre-MDT0000 complete [ 8167.270113] LustreError: 340291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388610 with bad export cookie 8663211983964663551 [ 8167.273479] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8167.273850] LustreError: 340291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8167.385643] Lustre: server umount lustre-MDT0001 complete [ 8177.878823] Lustre: server umount lustre-OST0000 complete [ 8183.027846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.180227] LustreError: 342266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8183.203490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8183.206212] Lustre: Skipped 1 previous similar message [ 8184.496466] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8187.516829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8188.999198] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8190.801550] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8192.026313] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8194.306073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8196.338705] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8198.098270] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8208.741963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8218.079878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8218.082287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8218.085459] Lustre: Skipped 2 previous similar messages [ 8218.087231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8218.089325] Lustre: Skipped 2 previous similar messages [ 8220.812435] Lustre: server umount lustre-OST0000 complete [ 8223.711789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8223.714443] LustreError: Skipped 1 previous similar message [ 8223.716614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8223.720657] Lustre: Skipped 1 previous similar message [ 8223.723161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8223.725215] Lustre: Skipped 2 previous similar messages [ 8228.181001] Lustre: server umount lustre-MDT0000 complete [ 8229.344105] LustreError: 342267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.349426] LustreError: 342267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8229.427198] LustreError: 343880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388672 with bad export cookie 8663211983964664559 [ 8229.430833] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8229.431205] LustreError: 343880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8229.547097] Lustre: server umount lustre-MDT0001 complete [ 8233.169962] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8234.245195] Key type lgssc unregistered [ 8234.366929] LNet: 345463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8234.370397] LNetError: 345463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8234.385307] LNet: Removed LNI 192.168.201.149@tcp [ 8234.692087] Key type .llcrypt unregistered [ 8234.693125] Key type ._llcrypt unregistered [ 8244.098458] Key type ._llcrypt registered [ 8244.100016] Key type .llcrypt registered [ 8244.137518] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8244.466047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8244.483090] alg: No test for adler32 (adler32-zlib) [ 8245.338222] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8245.423617] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8247.007127] Key type lgssc registered [ 8247.381313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8249.986603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8249.992310] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.995235] LustreError: 346862:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8250.002899] LustreError: 346862:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8250.004996] LustreError: 346862:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8250.007423] LustreError: 346862:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8250.010061] LustreError: 346862:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8252.531705] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.534749] LustreError: 347259:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8252.543853] LustreError: 347259:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8252.545949] LustreError: 347259:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8252.547963] LustreError: 347259:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8252.550431] LustreError: 347259:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8258.471028] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8259.420729] Key type lgssc unregistered [ 8259.543362] LNet: 348244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8259.546269] LNetError: 348244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8259.553262] LNet: Removed LNI 192.168.201.149@tcp [ 8259.824582] Key type .llcrypt unregistered [ 8259.825610] Key type ._llcrypt unregistered [ 8265.455952] Key type ._llcrypt registered [ 8265.457326] Key type .llcrypt registered [ 8265.493761] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8271.184109] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8271.444142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8271.449656] alg: No test for adler32 (adler32-zlib) [ 8272.284746] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8272.356677] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8273.927117] Key type lgssc registered [ 8274.270438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8284.664470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8284.668648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.747414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8285.757110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8285.791328] Lustre: lustre-MDT0000: new disk, initializing [ 8285.813834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.819759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8286.978460] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8291.080566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8291.124883] Lustre: 351344:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8291.150156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8291.152026] Lustre: Skipped 1 previous similar message [ 8291.187890] Lustre: lustre-MDT0001: new disk, initializing [ 8291.224968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8291.236812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8291.240184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8292.549646] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8295.650272] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8296.859513] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8299.128194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8299.236442] Lustre: lustre-OST0000: new disk, initializing [ 8299.239439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8299.244160] Lustre: 353085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8299.279488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8301.197899] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8303.288985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8303.291869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8303.321857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8304.152689] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8305.405815] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8305.470388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8306.626288] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8306.688659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8308.705186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8308.706286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.707780] LustreError: Skipped 1 previous similar message [ 8308.708512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8308.712415] Lustre: Skipped 1 previous similar message [ 8308.716602] Lustre: Skipped 1 previous similar message [ 8309.329781] Lustre: server umount lustre-OST0000 complete [ 8311.775793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8311.777915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.781281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8313.824033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8313.824603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8316.631506] Lustre: server umount lustre-MDT0000 complete [ 8317.862450] LustreError: 351339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388761 with bad export cookie 979025568540437390 [ 8317.865695] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.866322] LustreError: 351339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8317.981364] Lustre: server umount lustre-MDT0001 complete [ 8320.747851] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8321.275169] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:26:03 (1780388763) [ 8323.329988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.489379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8324.666197] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8327.282042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.575982] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8330.175230] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8331.250294] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8333.303859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8333.405876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.408650] Lustre: Skipped 1 previous similar message [ 8335.116221] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8336.715919] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8338.781371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.843522] Lustre: lustre-OST0001: new disk, initializing [ 8338.845695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8338.847962] Lustre: 357889:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8340.688837] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8343.615913] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8345.586557] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8345.588284] Lustre: Skipped 1 previous similar message [ 8350.200652] Lustre: lustre-OST0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 8350.202551] Lustre: Skipped 1 previous similar message [ 8351.414247] Lustre: server umount lustre-OST0001 complete [ 8355.811024] LustreError: 356987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.815914] LustreError: 356987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8360.930460] LustreError: 358484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8360.935205] LustreError: 358484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8363.488716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8363.491246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.496094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8363.498909] Lustre: Skipped 2 previous similar messages [ 8365.535882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8366.050845] LustreError: 356987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.056507] LustreError: 356987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8369.356651] Lustre: server umount lustre-OST0000 complete [ 8371.168722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8371.171674] Lustre: Skipped 1 previous similar message [ 8376.723127] Lustre: server umount lustre-MDT0000 complete [ 8377.844216] LustreError: 355376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388821 with bad export cookie 979025568540438258 [ 8377.844970] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.847805] LustreError: 355376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8392.159122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8392.233853] Lustre: server umount lustre-MDT0001 complete [ 8395.395817] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8396.319096] Key type lgssc unregistered [ 8396.439304] LNet: 359742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8396.441874] LNetError: 359742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8396.451294] LNet: Removed LNI 192.168.201.149@tcp [ 8396.737139] Key type .llcrypt unregistered [ 8396.738412] Key type ._llcrypt unregistered [ 8413.075695] Key type ._llcrypt registered [ 8413.077278] Key type .llcrypt registered [ 8413.112413] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8413.556579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8413.569598] alg: No test for adler32 (adler32-zlib) [ 8414.422288] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8414.500368] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8416.079157] Key type lgssc registered [ 8416.416393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8418.816337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8418.821713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8419.908477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8419.917164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8419.977377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.175843] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8423.780959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8423.809886] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8423.819230] Lustre: 362495:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8423.912545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8425.159358] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8426.845066] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8427.917618] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8429.023756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8429.026162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.030020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8431.583997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8431.584661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8434.638790] Lustre: server umount lustre-MDT0000 complete [ 8435.734553] LustreError: 362490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388878 with bad export cookie 10455029947541095138 [ 8435.737751] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8435.738555] LustreError: 362490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8435.840918] Lustre: server umount lustre-MDT0001 complete [ 8438.557988] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:28:00 (1780388880) [ 8440.594260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.767693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8442.052333] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8444.876977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8444.998734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8446.271492] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8447.996951] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8449.120303] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8451.302212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8451.371117] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8451.433281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8453.226412] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8454.895051] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8458.586366] Lustre: Permanently deactivating lustre-OST0000 [ 8458.592374] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8462.256977] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8463.139435] LustreError: 367375:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8463.145086] LustreError: 367375:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 8463.148867] LustreError: 367375:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8463.749214] LustreError: 367423:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 8463.751472] LustreError: 367423:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8470.479340] Lustre: server umount lustre-OST0000 complete [ 8471.386750] LustreError: 367673:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 8471.390151] LustreError: 367673:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8475.104849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8475.105614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8475.109636] Lustre: Skipped 1 previous similar message [ 8475.114165] Lustre: Skipped 2 previous similar messages [ 8480.224507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8480.227629] Lustre: Skipped 1 previous similar message [ 8485.344148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.346044] Lustre: Skipped 1 previous similar message [ 8486.367131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8486.427518] Lustre: server umount lustre-MDT0000 complete [ 8487.667200] LustreError: 364761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780388930 with bad export cookie 10455029947541095705 [ 8487.670222] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8487.671311] LustreError: 364761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8497.176177] Lustre: server umount lustre-MDT0001 complete [ 8500.120132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8501.481206] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8502.329629] LustreError: 368813:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8504.262646] Lustre: 369151:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.149@tcp [ 8504.904687] Lustre: server umount MGS complete [ 8508.445773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.578787] LustreError: 369874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8508.582797] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8508.583766] LustreError: 369874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8508.585493] Lustre: Skipped 1 previous similar message [ 8508.605772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8509.808465] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8512.465358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8512.556593] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8513.768458] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8515.416483] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8516.501645] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8517.110155] Lustre: Permanently reactivating lustre-OST0000 [ 8517.114166] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8517.115958] Lustre: Skipped 2 previous similar messages [ 8540.127114] LustreError: 371338:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8542.578897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8542.684316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8542.686629] Lustre: Skipped 1 previous similar message [ 8544.398585] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8545.976995] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8559.072105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8559.074412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8559.078615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8560.097671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8563.975812] Lustre: server umount lustre-OST0000 complete [ 8565.216185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8565.219184] Lustre: Skipped 1 previous similar message [ 8571.213475] Lustre: server umount lustre-MDT0000 complete [ 8572.345179] LustreError: 371221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389015 with bad export cookie 10455029947541096741 [ 8572.347894] LustreError: 371221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8572.348040] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8572.452616] Lustre: server umount lustre-MDT0001 complete [ 8575.606743] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8576.567037] Key type lgssc unregistered [ 8576.683294] LNet: 374093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8576.685572] LNetError: 374093:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8576.692301] LNet: Removed LNI 192.168.201.149@tcp [ 8576.973354] Key type .llcrypt unregistered [ 8576.974495] Key type ._llcrypt unregistered [ 8582.289105] Key type ._llcrypt registered [ 8582.290076] Key type .llcrypt registered [ 8582.325313] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8587.836253] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8588.134114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8588.161052] alg: No test for adler32 (adler32-zlib) [ 8589.002712] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8589.076295] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8590.647100] Key type lgssc registered [ 8591.010818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8602.154963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8602.160641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.253787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8603.267324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8603.308693] Lustre: lustre-MDT0000: new disk, initializing [ 8603.336089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8603.342802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8604.641740] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8608.778769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.820864] Lustre: 377187:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8608.843248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8608.844816] Lustre: Skipped 1 previous similar message [ 8608.879152] Lustre: lustre-MDT0001: new disk, initializing [ 8608.909653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8608.923087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8608.927249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8610.151612] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8613.160084] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8614.299461] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8616.496330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8616.594879] Lustre: lustre-OST0000: new disk, initializing [ 8616.597260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8616.601167] Lustre: 378928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8616.632229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8618.456613] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8621.558833] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8622.582679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8622.585823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8622.629228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8622.984836] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8623.067921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8624.399337] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8624.469875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8627.682637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8627.682809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8627.685026] LustreError: Skipped 1 previous similar message [ 8627.692787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8627.695126] Lustre: Skipped 1 previous similar message [ 8631.259933] Lustre: server umount lustre-OST0000 complete [ 8632.800374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8632.800907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8632.805151] Lustre: Skipped 2 previous similar messages [ 8637.920076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8637.922398] Lustre: Skipped 2 previous similar messages [ 8638.678582] Lustre: server umount lustre-MDT0000 complete [ 8639.812039] LustreError: 377964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389083 with bad export cookie 17586723554493450547 [ 8639.815340] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8639.816065] LustreError: 377964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8639.931578] Lustre: server umount lustre-MDT0001 complete [ 8642.566967] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8643.101451] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:31:25 (1780389085) [ 8645.254168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8645.416698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.628086] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8649.284344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8650.574840] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8652.153665] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8653.234112] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8655.271495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.368535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8655.370498] Lustre: Skipped 1 previous similar message [ 8657.065243] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8658.675419] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8659.281410] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8660.716356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8664.545631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8664.546076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8664.548093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8664.550667] Lustre: Skipped 1 previous similar message [ 8667.976312] Lustre: server umount lustre-OST0000 complete [ 8669.664213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.668339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8669.671228] Lustre: Skipped 2 previous similar messages [ 8675.284643] Lustre: server umount lustre-MDT0000 complete [ 8676.471060] LustreError: 381216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389119 with bad export cookie 17586723554493451415 [ 8676.473483] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8676.474616] LustreError: 381216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8676.574877] Lustre: server umount lustre-MDT0001 complete [ 8679.727559] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8680.644264] Key type lgssc unregistered [ 8680.769339] LNet: 384599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8680.771349] LNetError: 384599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8680.784280] LNet: Removed LNI 192.168.201.149@tcp [ 8681.055330] Key type .llcrypt unregistered [ 8681.056748] Key type ._llcrypt unregistered [ 8687.947589] Key type ._llcrypt registered [ 8687.948717] Key type .llcrypt registered [ 8687.978807] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8688.064565] Key type .llcrypt unregistered [ 8688.065639] Key type ._llcrypt unregistered [ 8694.735142] Key type ._llcrypt registered [ 8694.736330] Key type .llcrypt registered [ 8694.775696] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8695.143681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8695.182983] alg: No test for adler32 (adler32-zlib) [ 8696.056393] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8696.160338] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8697.751207] Key type lgssc registered [ 8698.137722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8700.807551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8700.813569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8701.984485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8703.351018] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8706.208804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8706.329685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.238369] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8709.909055] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8711.648434] LustreError: 386752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8711.654018] LustreError: 386752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8711.965167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8712.074321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8713.125493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8713.262330] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8717.280111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8717.282063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.285488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8718.305660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.306931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8718.307396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8718.314264] Lustre: Skipped 1 previous similar message [ 8721.992451] Lustre: server umount lustre-OST0000 complete [ 8723.424080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8723.424815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8723.428011] Lustre: Skipped 1 previous similar message [ 8723.430412] Lustre: Skipped 1 previous similar message [ 8728.544145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8728.546872] Lustre: Skipped 1 previous similar message [ 8729.296720] Lustre: server umount lustre-MDT0000 complete [ 8730.415155] LustreError: 388999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389173 with bad export cookie 11094962640346140200 [ 8730.418084] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8730.418252] LustreError: 388999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8730.523554] Lustre: server umount lustre-MDT0001 complete [ 8733.605261] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8734.546620] Key type lgssc unregistered [ 8734.671267] LNet: 389734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8734.673483] LNetError: 389734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8734.681287] LNet: Removed LNI 192.168.201.149@tcp [ 8734.973604] Key type .llcrypt unregistered [ 8734.974714] Key type ._llcrypt unregistered [ 8743.282341] Key type ._llcrypt registered [ 8743.283554] Key type .llcrypt registered [ 8743.315585] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8743.694151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8743.699111] alg: No test for adler32 (adler32-zlib) [ 8744.546204] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8744.614621] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8746.191155] Key type lgssc registered [ 8746.519915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8748.941994] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8748.948393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8764.383285] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8769.503135] LustreError: 391083:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.149@tcp: failed processing log, type 4: rc = -110 [ 8796.127170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.907858] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8799.541893] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8801.530186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.686522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8801.709836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8802.835712] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8805.472378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8805.580514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8806.730926] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8808.303503] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8809.333185] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8816.096493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8816.099715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.104167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8817.120432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8817.121424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.121441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8821.647406] Lustre: server umount lustre-OST0000 complete [ 8826.335946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8826.339041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.344066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8826.346017] Lustre: Skipped 1 previous similar message [ 8829.017098] Lustre: server umount lustre-MDT0000 complete [ 8830.311135] LustreError: 393272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389273 with bad export cookie 14101951177907542890 [ 8830.315092] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8830.316402] LustreError: 393272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8830.427654] Lustre: server umount lustre-MDT0001 complete [ 8834.309252] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8835.333672] Key type lgssc unregistered [ 8835.459310] LNet: 394295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8835.461572] LNetError: 394295:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8835.467313] LNet: Removed LNI 192.168.201.149@tcp [ 8835.789276] Key type .llcrypt unregistered [ 8835.791073] Key type ._llcrypt unregistered [ 8844.434876] Key type ._llcrypt registered [ 8844.435745] Key type .llcrypt registered [ 8844.466662] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8844.748512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8844.799272] alg: No test for adler32 (adler32-zlib) [ 8845.636961] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8845.711970] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8847.287130] Key type lgssc registered [ 8847.581886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.807704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8849.812724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.960819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8852.265299] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8855.007060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.133694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8856.446033] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8858.195353] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8859.332354] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8861.593039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8861.665241] LustreError: 397264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.673341] LustreError: 397264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8861.728460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8863.745128] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8864.807236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8865.640168] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8869.857244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8869.859554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8869.862065] Lustre: Skipped 2 previous similar messages [ 8869.863843] Lustre: Skipped 1 previous similar message [ 8871.927440] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 8871.929884] Lustre: Skipped 1 previous similar message [ 8873.612492] Lustre: server umount lustre-MDT0000 complete [ 8874.288508] LustreError: 396389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.49@tcp arrived at 1780389317 with bad export cookie 756132279288635915 [ 8874.292758] LustreError: 396389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8874.871051] LustreError: 395651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389318 with bad export cookie 756132279288635033 [ 8874.874414] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.875322] LustreError: 395651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8874.976028] LustreError: 396381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8874.976494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8874.976534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8874.977346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8874.980266] LustreError: 396381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8880.095684] LustreError: 395669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8880.095982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8880.101453] LustreError: 395669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8880.972297] Lustre: server umount lustre-OST0000 complete [ 8883.015317] Lustre: server umount lustre-MDT0001 complete [ 8886.041570] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8886.933991] Key type lgssc unregistered [ 8887.049429] LNet: 398994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8887.052828] LNetError: 398994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8887.066244] LNet: Removed LNI 192.168.201.149@tcp [ 8887.313820] Key type .llcrypt unregistered [ 8887.314663] Key type ._llcrypt unregistered [ 8895.787366] Key type ._llcrypt registered [ 8895.788436] Key type .llcrypt registered [ 8895.825975] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8896.144892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8896.194394] alg: No test for adler32 (adler32-zlib) [ 8897.049134] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8897.128051] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8898.703153] Key type lgssc registered [ 8899.056975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8901.540650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8901.547199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8902.712367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.044706] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8906.941362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8907.089668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8908.459224] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8910.325623] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8911.544158] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8913.660163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8913.782767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8915.711822] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8918.336985] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8927.087710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8932.323352] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.337247] Lustre: Skipped 1 previous similar message [ 8932.339310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8932.346173] Lustre: Skipped 1 previous similar message [ 8934.914255] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 8937.441594] LustreError: 401096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.451788] LustreError: 401096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8937.559237] Lustre: server umount lustre-MDT0001 complete [ 8942.560968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8942.566283] LustreError: 400370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8942.567713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.570903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8946.727198] Lustre: server umount lustre-OST0000 complete [ 8947.681949] LustreError: 400371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.814510] Lustre: server umount lustre-MDT0000 complete [ 8954.344723] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8955.596353] Key type lgssc unregistered [ 8955.731671] LNet: 403716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8955.738419] LNetError: 403716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8955.747857] LNet: Removed LNI 192.168.201.149@tcp [ 8956.108214] Key type .llcrypt unregistered [ 8956.109269] Key type ._llcrypt unregistered [ 8965.872172] Key type ._llcrypt registered [ 8965.873496] Key type .llcrypt registered [ 8965.915688] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8974.352739] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8975.107671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8975.147765] alg: No test for adler32 (adler32-zlib) [ 8976.102197] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 8976.238509] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8977.855132] Key type lgssc registered [ 8978.536299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8994.960340] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9000.957675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9000.963060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9002.052816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9002.066537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9002.106908] Lustre: lustre-MDT0000: new disk, initializing [ 9002.133512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9002.140202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9003.581263] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9009.485217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.614345] Lustre: lustre-OST0000: new disk, initializing [ 9009.616426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9009.620840] Lustre: Skipped 1 previous similar message [ 9009.624524] Lustre: 408658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9009.659729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.781584] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9015.140989] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9015.805403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9015.808491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9015.817762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9022.134954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.181850] Lustre: 407658:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9022.208251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9022.245273] Lustre: lustre-MDT0001: new disk, initializing [ 9022.289667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9022.302794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9022.306079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9023.699918] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9027.240298] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9027.306686] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9028.614816] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9031.739701] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 9035.260746] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 9036.769549] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9036.771249] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9036.773189] Lustre: Skipped 1 previous similar message [ 9036.776503] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9039.447953] Lustre: server umount lustre-MDT0001 complete [ 9047.012074] LustreError: 409885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9047.018679] LustreError: 409885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9052.129206] LustreError: 407670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9052.130341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9052.142059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9052.144447] Lustre: Skipped 1 previous similar message [ 9057.250865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9057.251863] LustreError: 409885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9058.231390] Lustre: server umount lustre-OST0000 complete [ 9066.198367] Lustre: server umount lustre-MDT0000 complete [ 9071.491216] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9072.703058] Key type lgssc unregistered [ 9072.849347] LNet: 412246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9072.852246] LNetError: 412246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9072.860318] LNet: Removed LNI 192.168.201.149@tcp [ 9073.231105] Key type .llcrypt unregistered [ 9073.232711] Key type ._llcrypt unregistered [ 9089.526768] Key type ._llcrypt registered [ 9089.528151] Key type .llcrypt registered [ 9089.583963] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9089.718315] Key type .llcrypt unregistered [ 9089.719843] Key type ._llcrypt unregistered [ 9097.133818] Key type ._llcrypt registered [ 9097.135219] Key type .llcrypt registered [ 9097.180118] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9097.545251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9097.574052] alg: No test for adler32 (adler32-zlib) [ 9098.448624] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [ 9098.538631] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 9100.127147] Key type lgssc registered [ 9100.655831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9112.452644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9112.468857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9113.595790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9113.613062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9113.654135] Lustre: lustre-MDT0000: new disk, initializing [ 9113.694341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9113.707332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9115.308542] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9120.393529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9120.458801] Lustre: 415561:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9120.498590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9120.500894] Lustre: Skipped 1 previous similar message [ 9120.541396] Lustre: lustre-MDT0001: new disk, initializing [ 9120.585753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9120.603253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9120.608174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9122.339527] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9125.859614] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9127.288813] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9129.994118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9130.108460] Lustre: lustre-OST0000: new disk, initializing [ 9130.110775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9130.113868] Lustre: 417303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9130.154973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9132.516952] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9135.969879] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9136.646355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9136.649907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9136.694290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9146.337470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9146.341543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9146.347974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9146.850107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9146.850956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9146.857432] Lustre: Skipped 1 previous similar message [ 9146.866458] Lustre: Skipped 1 previous similar message [ 9149.916164] Lustre: server umount lustre-MDT0000 complete [ 9151.401393] LustreError: 416339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389594 with bad export cookie 5040280365514918201 [ 9151.402230] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9151.405029] LustreError: 416339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9151.530736] Lustre: server umount lustre-MDT0001 complete [ 9163.230828] Lustre: server umount lustre-OST0000 complete [ 9167.366199] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:40:09 (1780389609) [ 9170.460191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.640837] LustreError: 419216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.648652] LustreError: 419216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9170.668182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9172.178589] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9175.377284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.532407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9176.999143] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9179.154244] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9180.542113] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9196.068131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9196.168691] Lustre: Found index 0 for lustre-OST0000, updating log [ 9196.256172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9198.569782] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9200.567629] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9216.744582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9226.208260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9226.211838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.218066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9229.201941] Lustre: server umount lustre-OST0000 complete [ 9242.079749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9242.081797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.085379] Lustre: Skipped 1 previous similar message [ 9242.087425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.089461] Lustre: Skipped 3 previous similar messages [ 9248.024040] Lustre: server umount lustre-MDT0000 complete [ 9248.736344] LustreError: 419216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.743191] LustreError: 419216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9249.362299] LustreError: 419196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780389692 with bad export cookie 5040280365514919601 [ 9249.365218] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9249.368365] LustreError: 419196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9249.478139] Lustre: server umount lustre-MDT0001 complete [ 9252.629575] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:41:34 (1780389694) [ 9259.295291] LNet: 1 local NIs in recovery (showing 1): 192.168.201.149@tcp [ 9844.360162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9844.636035] LustreError: 425302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.694831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9846.824916] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9849.824256] LustreError: 425303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.442712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.643212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9853.689570] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9856.617423] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9858.777746] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9875.811899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9875.952882] Lustre: Found index 0 for lustre-OST0000, updating log [ 9876.144743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.889702] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9883.262647] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9888.736162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9888.738814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9888.742925] Lustre: Skipped 1 previous similar message [ 9888.745821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9888.748350] Lustre: Skipped 2 previous similar messages [ 9890.479880] Lustre: server umount lustre-OST0000 complete [ 9893.923652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.932744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9893.936550] Lustre: Skipped 1 previous similar message [ 9894.945939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9896.993231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9896.998012] Lustre: Skipped 4 previous similar messages [ 9898.271875] LNet: 1 local NIs in recovery (showing 1): 192.168.201.149@tcp [ 9901.088984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9901.091662] Lustre: Skipped 6 previous similar messages [ 9907.167105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9907.278354] Lustre: server umount lustre-MDT0000 complete [ 9908.259281] LustreError: 425298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9908.281976] LustreError: 425298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9909.734565] LustreError: 425285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390352 with bad export cookie 5040280365514920672 [ 9909.736613] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9909.747535] LustreError: 425285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9909.936878] Lustre: server umount lustre-MDT0001 complete [ 9932.984143] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 9938.766496] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9959.231503] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:53:20 (1780390400) [ 9968.134491] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9969.861398] Key type lgssc unregistered [ 9970.035950] LNet: 431516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9970.043147] LNetError: 431516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10007.974123] LNet: Removed LNI 192.168.201.149@tcp [10008.377199] Key type .llcrypt unregistered [10008.378182] Key type ._llcrypt unregistered [10017.230552] Key type ._llcrypt registered [10017.232516] Key type .llcrypt registered [10017.305657] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10017.925289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10017.935058] alg: No test for adler32 (adler32-zlib) [10018.845461] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [10019.001692] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10020.599510] Key type lgssc registered [10021.178952] Lustre: Echo OBD driver; http://www.lustre.org/ [10023.379147] LNet: Added LNI 192.168.201.149@tcp100 [8/256/0/180] [10023.968346] LNet: Added LNI 192.168.201.149@tcp101 [8/256/0/180] [10024.456479] LNet: Added LNI 192.168.201.149@tcp102 [8/256/0/180] [10024.911151] LNet: Added LNI 192.168.201.149@tcp103 [8/256/0/180] [10025.360166] LNet: Added LNI 192.168.201.149@tcp104 [8/256/0/180] [10025.838958] LNet: Added LNI 192.168.201.149@tcp105 [8/256/0/180] [10026.285782] LNet: Added LNI 192.168.201.149@tcp106 [8/256/0/180] [10026.773365] LNet: Added LNI 192.168.201.149@tcp107 [8/256/0/180] [10027.204718] LNet: Added LNI 192.168.201.149@tcp108 [8/256/0/180] [10027.566689] LNet: Added LNI 192.168.201.149@tcp109 [8/256/0/180] [10027.917331] LNet: Added LNI 192.168.201.149@tcp110 [8/256/0/180] [10028.326942] LNet: Added LNI 192.168.201.149@tcp111 [8/256/0/180] [10028.712475] LNet: Added LNI 192.168.201.149@tcp112 [8/256/0/180] [10029.204830] LNet: Added LNI 192.168.201.149@tcp113 [8/256/0/180] [10029.632563] LNet: Added LNI 192.168.201.149@tcp114 [8/256/0/180] [10030.088351] LNet: Added LNI 192.168.201.149@tcp115 [8/256/0/180] [10030.506244] LNet: Added LNI 192.168.201.149@tcp116 [8/256/0/180] [10030.905358] LNet: Added LNI 192.168.201.149@tcp117 [8/256/0/180] [10031.291074] LNet: Added LNI 192.168.201.149@tcp118 [8/256/0/180] [10031.658668] LNet: Added LNI 192.168.201.149@tcp119 [8/256/0/180] [10032.027523] LNet: Added LNI 192.168.201.149@tcp120 [8/256/0/180] [10032.444511] LNet: Added LNI 192.168.201.149@tcp121 [8/256/0/180] [10032.933671] LNet: Added LNI 192.168.201.149@tcp122 [8/256/0/180] [10033.403340] LNet: Added LNI 192.168.201.149@tcp123 [8/256/0/180] [10033.854561] LNet: Added LNI 192.168.201.149@tcp124 [8/256/0/180] [10034.334924] LNet: Added LNI 192.168.201.149@tcp125 [8/256/0/180] [10034.765847] LNet: Added LNI 192.168.201.149@tcp126 [8/256/0/180] [10035.227357] LNet: Added LNI 192.168.201.149@tcp127 [8/256/0/180] [10035.715699] LNet: Added LNI 192.168.201.149@tcp128 [8/256/0/180] [10036.180140] LNet: Added LNI 192.168.201.149@tcp129 [8/256/0/180] [10036.589859] LNet: Added LNI 192.168.201.149@tcp130 [8/256/0/180] [10036.989253] LNet: Added LNI 192.168.201.149@tcp131 [8/256/0/180] [10037.444210] LNet: Added LNI 192.168.201.149@tcp132 [8/256/0/180] [10037.842373] LNet: Added LNI 192.168.201.149@tcp133 [8/256/0/180] [10038.352744] LNet: Added LNI 192.168.201.149@tcp134 [8/256/0/180] [10038.881121] LNet: Added LNI 192.168.201.149@tcp135 [8/256/0/180] [10042.710944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10042.730536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10043.836964] Lustre: 434493:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [10043.930145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10043.965219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10044.008787] Lustre: lustre-MDT0000: new disk, initializing [10044.061976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10044.072666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10046.664667] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10052.912056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.985194] Lustre: 434510:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10053.015552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10053.017410] Lustre: Skipped 1 previous similar message [10053.063400] Lustre: lustre-MDT0001: new disk, initializing [10053.085606] Lustre: 435293:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [10053.091873] Lustre: 435293:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10053.167678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10053.197233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10053.208145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10055.218927] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10058.256332] Lustre: 435714:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [10058.266646] Lustre: 435714:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10059.699752] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10061.818679] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10065.364339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10065.515667] Lustre: lustre-OST0000: new disk, initializing [10065.517392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10065.520860] Lustre: 436257:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10065.573230] Lustre: 436262:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [10065.578630] Lustre: 436262:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [10065.605412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10068.229786] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10072.091541] Lustre: 436806:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [10072.101555] Lustre: 436806:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10072.157472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10072.161255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10072.220392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10072.476623] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10077.666714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10077.667281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10077.670602] LustreError: Skipped 1 previous similar message [10077.677263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10077.680652] Lustre: Skipped 1 previous similar message [10078.370467] Lustre: server umount lustre-OST0000 complete [10082.785795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.789149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10082.792013] Lustre: Skipped 2 previous similar messages [10086.282972] Lustre: server umount lustre-MDT0000 complete [10088.239703] LustreError: 435292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390531 with bad export cookie 2750195264375151227 [10088.240439] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10088.244831] LustreError: 435292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10088.390932] Lustre: server umount lustre-MDT0001 complete [10093.250616] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10094.607916] Key type lgssc unregistered [10094.841178] LNet: 438037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10094.847126] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10094.850678] LNet: Removed LNI 192.168.201.149@tcp [10095.904270] LNet: Removed LNI 192.168.201.149@tcp100 [10095.906178] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10095.913226] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10095.922903] LNet: Removed LNI 192.168.201.149@tcp101 [10096.927160] LNet: Removed LNI 192.168.201.149@tcp102 [10096.929582] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10096.934653] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10096.937368] LNet: Removed LNI 192.168.201.149@tcp103 [10097.951153] LNet: Removed LNI 192.168.201.149@tcp104 [10098.975143] LNet: Removed LNI 192.168.201.149@tcp105 [10098.977127] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10098.982085] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10098.986104] LNet: Removed LNI 192.168.201.149@tcp106 [10099.999130] LNet: Removed LNI 192.168.201.149@tcp107 [10101.023125] LNet: Removed LNI 192.168.201.149@tcp108 [10102.047184] LNet: Removed LNI 192.168.201.149@tcp109 [10103.071124] LNet: Removed LNI 192.168.201.149@tcp110 [10103.073683] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10103.079816] LNetError: 438037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10103.082702] LNet: Removed LNI 192.168.201.149@tcp111 [10103.083856] LNet: Removed LNI 192.168.201.149@tcp112 [10103.085758] LNet: Removed LNI 192.168.201.149@tcp113 [10103.087099] LNet: Removed LNI 192.168.201.149@tcp114 [10103.088908] LNet: Removed LNI 192.168.201.149@tcp115 [10103.090823] LNet: Removed LNI 192.168.201.149@tcp116 [10103.092542] LNet: Removed LNI 192.168.201.149@tcp117 [10103.093892] LNet: Removed LNI 192.168.201.149@tcp118 [10103.095337] LNet: Removed LNI 192.168.201.149@tcp119 [10103.096714] LNet: Removed LNI 192.168.201.149@tcp120 [10103.098540] LNet: Removed LNI 192.168.201.149@tcp121 [10103.100675] LNet: Removed LNI 192.168.201.149@tcp122 [10103.102633] LNet: Removed LNI 192.168.201.149@tcp123 [10103.104548] LNet: Removed LNI 192.168.201.149@tcp124 [10103.106041] LNet: Removed LNI 192.168.201.149@tcp125 [10103.107340] LNet: Removed LNI 192.168.201.149@tcp126 [10103.108561] LNet: Removed LNI 192.168.201.149@tcp127 [10103.110082] LNet: Removed LNI 192.168.201.149@tcp128 [10103.111824] LNet: Removed LNI 192.168.201.149@tcp129 [10103.113371] LNet: Removed LNI 192.168.201.149@tcp130 [10103.114987] LNet: Removed LNI 192.168.201.149@tcp131 [10103.119907] LNet: Removed LNI 192.168.201.149@tcp132 [10103.121319] LNet: Removed LNI 192.168.201.149@tcp133 [10103.122708] LNet: Removed LNI 192.168.201.149@tcp134 [10103.133124] LNet: Removed LNI 192.168.201.149@tcp135 [10103.475818] Key type .llcrypt unregistered [10103.477232] Key type ._llcrypt unregistered [10115.082551] Key type ._llcrypt registered [10115.083452] Key type .llcrypt registered [10115.138944] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [10123.273657] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10123.901561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10123.926849] alg: No test for adler32 (adler32-zlib) [10124.874048] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [10125.012287] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10126.631243] Key type lgssc registered [10127.306672] Lustre: Echo OBD driver; http://www.lustre.org/ [10146.465862] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:56:28 (1780390588) [10155.417966] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10156.859209] Key type lgssc unregistered [10157.038775] LNet: 442044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10157.043712] LNetError: 442044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10157.061408] LNet: Removed LNI 192.168.201.149@tcp [10157.417609] Key type .llcrypt unregistered [10157.419029] Key type ._llcrypt unregistered [10167.115486] Key type ._llcrypt registered [10167.117249] Key type .llcrypt registered [10167.181510] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10167.839218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10167.869389] alg: No test for adler32 (adler32-zlib) [10168.760966] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [10168.863985] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10170.487330] Key type lgssc registered [10171.069693] Lustre: Echo OBD driver; http://www.lustre.org/ [10176.206779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10176.223228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10177.349610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10177.367417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10177.415774] Lustre: lustre-MDT0000: new disk, initializing [10177.475627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10177.491292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10179.372752] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10185.332907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10185.384162] Lustre: 443219:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10185.414348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10185.418309] Lustre: Skipped 1 previous similar message [10185.469760] Lustre: lustre-MDT0001: new disk, initializing [10185.529503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10185.553378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10185.562288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10187.564487] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10192.142198] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10194.157627] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10197.733963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10197.866363] Lustre: lustre-OST0000: new disk, initializing [10197.869199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10197.875041] Lustre: 444961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10197.936543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10200.678832] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10204.678769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10204.685628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10204.740380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10204.799509] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10206.413434] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing lnet_if_list [10208.692209] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10208.707534] LNet: Added LNI 192.168.201.149@tcp10 [8/256/0/180] [10211.077023] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10211.089805] LNet: Added LNI 192.168.201.149@tcp11 [8/256/0/180] [10212.709395] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10212.721819] LNet: Added LNI 192.168.201.149@tcp12 [8/256/0/180] [10214.180309] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10214.195784] LNet: Added LNI 192.168.201.149@tcp13 [8/256/0/180] [10215.829636] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10215.851198] LNet: Added LNI 192.168.201.149@tcp14 [8/256/0/180] [10217.748564] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10217.767851] LNet: Added LNI 192.168.201.149@tcp15 [8/256/0/180] [10219.866792] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10219.880219] LNet: Added LNI 192.168.201.149@tcp16 [8/256/0/180] [10222.061238] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10222.079210] LNet: Added LNI 192.168.201.149@tcp17 [8/256/0/180] [10224.291760] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10224.309670] LNet: Added LNI 192.168.201.149@tcp18 [8/256/0/180] [10226.553423] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10226.567414] LNet: Added LNI 192.168.201.149@tcp19 [8/256/0/180] [10228.576962] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10228.591672] LNet: Added LNI 192.168.201.149@tcp20 [8/256/0/180] [10230.590178] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10230.605605] LNet: Added LNI 192.168.201.149@tcp21 [8/256/0/180] [10232.827367] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10232.842831] LNet: Added LNI 192.168.201.149@tcp22 [8/256/0/180] [10235.039650] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10235.052739] LNet: Added LNI 192.168.201.149@tcp23 [8/256/0/180] [10237.172889] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10237.189907] LNet: Added LNI 192.168.201.149@tcp24 [8/256/0/180] [10239.279432] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10239.293701] LNet: Added LNI 192.168.201.149@tcp25 [8/256/0/180] [10241.335821] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10241.348950] LNet: Added LNI 192.168.201.149@tcp26 [8/256/0/180] [10243.736369] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10243.750443] LNet: Added LNI 192.168.201.149@tcp27 [8/256/0/180] [10245.939589] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10245.954311] LNet: Added LNI 192.168.201.149@tcp28 [8/256/0/180] [10248.509486] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10248.527251] LNet: Added LNI 192.168.201.149@tcp29 [8/256/0/180] [10251.098116] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10251.112623] LNet: Added LNI 192.168.201.149@tcp30 [8/256/0/180] [10253.874694] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10253.889995] LNet: Added LNI 192.168.201.149@tcp31 [8/256/0/180] [10256.712735] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10256.726067] LNet: Added LNI 192.168.201.149@tcp32 [8/256/0/180] [10259.351474] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10259.366827] LNet: Added LNI 192.168.201.149@tcp33 [8/256/0/180] [10262.154624] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10262.167750] LNet: Added LNI 192.168.201.149@tcp34 [8/256/0/180] [10264.973423] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10264.994148] LNet: Added LNI 192.168.201.149@tcp35 [8/256/0/180] [10267.655813] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10267.671489] LNet: Added LNI 192.168.201.149@tcp36 [8/256/0/180] [10270.472139] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10270.491397] LNet: Added LNI 192.168.201.149@tcp37 [8/256/0/180] [10273.475953] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10273.495092] LNet: Added LNI 192.168.201.149@tcp38 [8/256/0/180] [10276.422501] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10276.434368] LNet: Added LNI 192.168.201.149@tcp39 [8/256/0/180] [10279.296650] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10279.313490] LNet: Added LNI 192.168.201.149@tcp40 [8/256/0/180] [10282.200400] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10282.224592] LNet: Added LNI 192.168.201.149@tcp41 [8/256/0/180] [10285.195126] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10285.206994] LNet: Added LNI 192.168.201.149@tcp42 [8/256/0/180] [10288.202610] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10288.218686] LNet: Added LNI 192.168.201.149@tcp43 [8/256/0/180] [10291.405853] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10291.423179] LNet: Added LNI 192.168.201.149@tcp44 [8/256/0/180] [10294.663687] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10294.686516] LNet: Added LNI 192.168.201.149@tcp45 [8/256/0/180] [10298.216575] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10298.233043] LNet: Added LNI 192.168.201.149@tcp46 [8/256/0/180] [10301.678064] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10301.694522] LNet: Added LNI 192.168.201.149@tcp47 [8/256/0/180] [10305.057682] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10305.071762] LNet: Added LNI 192.168.201.149@tcp48 [8/256/0/180] [10308.271705] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10308.285228] LNet: Added LNI 192.168.201.149@tcp49 [8/256/0/180] [10311.572370] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10311.586452] LNet: Added LNI 192.168.201.149@tcp50 [8/256/0/180] [10322.404193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10322.408584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10322.419536] Lustre: Skipped 2 previous similar messages [10322.429337] Lustre: Skipped 2 previous similar messages [10323.139830] Lustre: server umount lustre-MDT0000 complete [10324.971321] LustreError: 443211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390768 with bad export cookie 5929215648010077176 [10324.979292] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10324.981747] LustreError: 443211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10325.118086] Lustre: server umount lustre-MDT0001 complete [10333.993264] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10338.266202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10343.839104] Lustre: 442493:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780390770/real 1780390770] req@000000009c118767 x1866874866462208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780390786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10343.849704] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10344.863379] Lustre: 442493:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780390772/real 1780390772] req@00000000444eac82 x1866874866462336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780390788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10347.999134] Lustre: 442490:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780390775/real 1780390775] req@0000000074dd2686 x1866874866462464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780390791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10349.983401] LustreError: 442489:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005628807d x1866874866463360/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10350.120598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10351.136089] Lustre: 442490:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780390778/real 1780390778] req@0000000030016dd9 x1866874866462720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780390794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10351.686334] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10355.444925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.638428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10357.303449] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10357.665084] Lustre: 442492:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780390784/real 1780390784] req@00000000ad54e71a x1866874866462848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780390800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10359.654484] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10360.806161] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10361.151786] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10362.855353] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10370.529976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10370.535701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10370.544925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10371.040896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10373.089769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10373.093295] Lustre: Skipped 2 previous similar messages [10375.990065] Lustre: server umount lustre-OST0000 complete [10378.209256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10378.210034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10378.214570] Lustre: Skipped 2 previous similar messages [10378.219949] Lustre: Skipped 1 previous similar message [10383.328524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10383.330428] Lustre: Skipped 1 previous similar message [10383.771246] Lustre: server umount lustre-MDT0000 complete [10385.460331] LustreError: 443211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390828 with bad export cookie 5929215648010079367 [10385.464232] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10385.468990] LustreError: 443211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10385.600889] Lustre: server umount lustre-MDT0001 complete [10389.793044] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10390.961325] Key type lgssc unregistered [10391.148867] LNet: 470839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10391.152405] LNetError: 470839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10391.156096] LNet: Removed LNI 192.168.201.149@tcp [10391.158025] LNet: Removed LNI 192.168.201.149@tcp10 [10391.159588] LNet: Removed LNI 192.168.201.149@tcp11 [10391.161051] LNet: Removed LNI 192.168.201.149@tcp12 [10391.162655] LNet: Removed LNI 192.168.201.149@tcp13 [10391.164033] LNet: Removed LNI 192.168.201.149@tcp14 [10391.165670] LNet: Removed LNI 192.168.201.149@tcp15 [10391.167101] LNet: Removed LNI 192.168.201.149@tcp16 [10391.168577] LNet: Removed LNI 192.168.201.149@tcp17 [10391.170496] LNet: Removed LNI 192.168.201.149@tcp18 [10391.172576] LNet: Removed LNI 192.168.201.149@tcp19 [10391.174586] LNet: Removed LNI 192.168.201.149@tcp20 [10391.176489] LNet: Removed LNI 192.168.201.149@tcp21 [10391.178274] LNet: Removed LNI 192.168.201.149@tcp22 [10391.180108] LNet: Removed LNI 192.168.201.149@tcp23 [10391.181908] LNet: Removed LNI 192.168.201.149@tcp24 [10391.183673] LNet: Removed LNI 192.168.201.149@tcp25 [10391.185535] LNet: Removed LNI 192.168.201.149@tcp26 [10391.187101] LNet: Removed LNI 192.168.201.149@tcp27 [10391.188572] LNet: Removed LNI 192.168.201.149@tcp28 [10391.190132] LNet: Removed LNI 192.168.201.149@tcp29 [10391.191676] LNet: Removed LNI 192.168.201.149@tcp30 [10391.193193] LNet: Removed LNI 192.168.201.149@tcp31 [10391.194712] LNet: Removed LNI 192.168.201.149@tcp32 [10391.196101] LNet: Removed LNI 192.168.201.149@tcp33 [10391.197724] LNet: Removed LNI 192.168.201.149@tcp34 [10391.199150] LNet: Removed LNI 192.168.201.149@tcp35 [10391.200578] LNet: Removed LNI 192.168.201.149@tcp36 [10391.202590] LNet: Removed LNI 192.168.201.149@tcp37 [10391.204477] LNet: Removed LNI 192.168.201.149@tcp38 [10391.206269] LNet: Removed LNI 192.168.201.149@tcp39 [10391.208100] LNet: Removed LNI 192.168.201.149@tcp40 [10391.209880] LNet: Removed LNI 192.168.201.149@tcp41 [10391.211728] LNet: Removed LNI 192.168.201.149@tcp42 [10391.213264] LNet: Removed LNI 192.168.201.149@tcp43 [10391.214852] LNet: Removed LNI 192.168.201.149@tcp44 [10391.217571] LNet: Removed LNI 192.168.201.149@tcp45 [10391.220493] LNet: Removed LNI 192.168.201.149@tcp46 [10391.222743] LNet: Removed LNI 192.168.201.149@tcp47 [10391.224821] LNet: Removed LNI 192.168.201.149@tcp48 [10391.226644] LNet: Removed LNI 192.168.201.149@tcp49 [10391.232404] LNet: Removed LNI 192.168.201.149@tcp50 [10391.566704] Key type .llcrypt unregistered [10391.568350] Key type ._llcrypt unregistered [10399.340364] Key type ._llcrypt registered [10399.342011] Key type .llcrypt registered [10399.386860] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [10406.008883] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10406.381869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10406.434304] alg: No test for adler32 (adler32-zlib) [10407.300485] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [10407.394294] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10408.983347] Key type lgssc registered [10409.387562] Lustre: Echo OBD driver; http://www.lustre.org/ [10422.207399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10422.213106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10423.307275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10423.320725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10423.365153] Lustre: lustre-MDT0000: new disk, initializing [10423.392311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10423.399991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10424.836561] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10429.232626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10429.279238] Lustre: 473986:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10429.303911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10429.306075] Lustre: Skipped 1 previous similar message [10429.343821] Lustre: lustre-MDT0001: new disk, initializing [10429.378384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10429.394420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10429.398338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10430.837191] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10434.081443] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10435.385594] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10438.204102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10438.281873] Lustre: *** cfs_fail_loc=90f, val=0*** [10438.312719] Lustre: lustre-OST0000: new disk, initializing [10438.314483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10438.317415] Lustre: 475777:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10438.348216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10439.543782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10439.547243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10439.581667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10440.416651] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10443.626199] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10449.889149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10449.891346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10449.892480] LustreError: Skipped 1 previous similar message [10449.898211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10449.902452] Lustre: Skipped 1 previous similar message [10452.309516] Lustre: server umount lustre-OST0000 complete [10455.008923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10455.009295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10455.010042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10455.013872] Lustre: Skipped 2 previous similar messages [10459.870649] Lustre: server umount lustre-MDT0000 complete [10461.246336] LustreError: 473977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780390904 with bad export cookie 10827806425813755945 [10461.249968] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10461.254145] LustreError: 473977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10461.376299] Lustre: server umount lustre-MDT0001 complete [10465.291043] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10466.410444] Key type lgssc unregistered [10466.546620] LNet: 477547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10466.549904] LNetError: 477547:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10466.558429] LNet: Removed LNI 192.168.201.149@tcp [10466.901732] Key type .llcrypt unregistered [10466.903707] Key type ._llcrypt unregistered [10474.840219] Key type ._llcrypt registered [10474.841485] Key type .llcrypt registered [10474.888279] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10475.369970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10475.377306] alg: No test for adler32 (adler32-zlib) [10476.253078] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [10476.339476] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10477.935166] Key type lgssc registered [10478.393843] Lustre: Echo OBD driver; http://www.lustre.org/ [10481.334497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10481.344157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.526264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10484.044944] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10487.247112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10487.391302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10488.850195] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10490.830582] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10492.250425] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10494.923183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10495.072493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10497.150118] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10499.078272] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10513.690647] loop0: detected capacity change from 0 to 204800000 [10513.753262] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10513.757523] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10513.762870] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10516.019158] loop0: detected capacity change from 0 to 204800000 [10516.090503] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10516.094657] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10516.099409] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10518.198329] loop0: detected capacity change from 0 to 204800000 [10520.376194] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10520.437360] Lustre: fs15246-OST0000: new disk, initializing [10520.441269] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10520.445809] Lustre: 482047:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10520.477995] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10522.647659] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10526.867653] loop1: detected capacity change from 0 to 204800000 [10529.048672] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10529.093650] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10529.121190] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10529.159737] Lustre: fs15246-MDT0000: new disk, initializing [10529.218034] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10529.228719] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10529.273825] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10530.335751] Lustre: 483151:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000012741a3f x1866875188313472/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780390989 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10530.704084] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10538.884577] LustreError: 483151:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [10538.888459] LustreError: 483151:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [10538.892048] LustreError: 483151:0:(osp_precreate.c:460:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [10543.587592] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10543.592592] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10545.038888] Lustre: server umount fs15246-MDT0000 complete [10548.539089] Lustre: server umount fs15246-OST0000 complete [10553.824790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10553.826230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.828054] LustreError: Skipped 1 previous similar message [10553.834026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10553.837576] Lustre: Skipped 1 previous similar message [10558.356854] Lustre: server umount lustre-OST0000 complete [10564.064542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10564.064966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10564.065636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10564.070062] Lustre: Skipped 2 previous similar messages [10569.184095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10569.187739] Lustre: Skipped 2 previous similar messages [10574.303311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10574.304483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10574.311531] Lustre: Skipped 1 previous similar message [10574.379547] Lustre: server umount lustre-MDT0000 complete [10575.774460] LustreError: 478758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780391018 with bad export cookie 1005093599957407262 [10575.774773] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10575.779105] LustreError: 478758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10575.902809] Lustre: server umount lustre-MDT0001 complete [10579.823088] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10580.921297] Key type lgssc unregistered [10581.060376] LNet: 486211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10581.063092] LNetError: 486211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10581.069308] LNet: Removed LNI 192.168.201.149@tcp [10581.388237] Key type .llcrypt unregistered [10581.389776] Key type ._llcrypt unregistered [10589.304465] Key type ._llcrypt registered [10589.305667] Key type .llcrypt registered [10589.350650] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10589.747807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10589.790067] alg: No test for adler32 (adler32-zlib) [10590.662140] Lustre: Lustre: Build Version: 2.17.53_57_ge5fd5c2 [10590.769667] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [10592.375258] Key type lgssc registered [10592.810727] Lustre: Echo OBD driver; http://www.lustre.org/ [10608.074077] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [10612.476093] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10626.777648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10626.786134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.895927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10627.914789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10627.955160] Lustre: lustre-MDT0000: new disk, initializing [10627.983330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10627.991231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10629.593337] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10634.370297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10634.425622] Lustre: 490595:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10634.453703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10634.455762] Lustre: Skipped 1 previous similar message [10634.496258] Lustre: lustre-MDT0001: new disk, initializing [10634.538441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10634.554655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10634.558222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10636.019708] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10639.391653] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10640.736540] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10643.313363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10643.418645] Lustre: lustre-OST0000: new disk, initializing [10643.421063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10643.424318] Lustre: 492339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10643.456653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10645.629675] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [10648.581684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10648.585550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10648.623400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10649.020964] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10650.569677] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10650.652376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10652.135406] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10652.213875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10653.665860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10653.666572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10653.669070] LustreError: Skipped 1 previous similar message [10653.674156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10653.677837] Lustre: Skipped 1 previous similar message [10655.007821] Lustre: server umount lustre-OST0000 complete [10658.785874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.786613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10658.790665] Lustre: Skipped 2 previous similar messages [10662.555390] Lustre: server umount lustre-MDT0000 complete [10663.785913] LustreError: 490587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780391106 with bad export cookie 11133221543479303375 [10663.789726] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10663.790202] LustreError: 490587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10663.913058] Lustre: server umount lustre-MDT0001 complete [10672.151769] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [10675.978208] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [10688.131284] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10381 sec ========= 05:05:29 (1780391129) [10688.740440] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:05:30 (1780391130) === [10689.969665] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:05:31 (1780391131) === [10703.886946] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [10704.966702] Key type lgssc unregistered [10705.096381] LNet: 498770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10705.099461] LNetError: 498770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10705.107585] LNet: Removed LNI 192.168.201.149@tcp [10705.424314] Key type .llcrypt unregistered [10705.426088] Key type ._llcrypt unregistered