[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514270784 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002381] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007536] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011100] LSM: Security Framework initializing [ 0.012043] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014056] *** VALIDATE selinux *** [ 0.021724] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026494] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] *** VALIDATE tmpfs *** [ 0.031251] *** VALIDATE proc *** [ 0.032200] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037035] Spectre V2 : User space: Vulnerable [ 0.038020] Speculative Store Bypass: Vulnerable [ 0.041356] debug: unmapping init [mem 0xffffffffbd459000-0xffffffffbd460fff] [ 0.043169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044651] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.045996] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051342] rcu: Hierarchical SRCU implementation. [ 0.053500] smp: Bringing up secondary CPUs ... [ 0.054544] x86: Booting SMP configuration: [ 0.055024] .... node #0, CPUs: #1 #2 #3 [ 0.058105] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137674] node 0 deferred pages initialised in 74ms [ 0.141135] devtmpfs: initialized [ 0.142210] x86/mm: Memory block size: 128MB [ 0.144798] gcov: version magic: 0x41383552 [ 0.146145] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151339] pinctrl core: initialized pinctrl subsystem [ 0.153167] [ 0.153673] ************************************************************* [ 0.156015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158009] ** ** [ 0.159007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161013] ** ** [ 0.163013] ** This means that this kernel is built to expose internal ** [ 0.165015] ** IOMMU data structures, which may compromise security on ** [ 0.167013] ** your system. ** [ 0.170012] ** ** [ 0.172011] ** If you see this message and you are not debugging the ** [ 0.174014] ** kernel, report this immediately to your vendor! ** [ 0.176015] ** ** [ 0.179012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181012] ************************************************************* [ 0.184639] NET: Registered protocol family 16 [ 0.186456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193460] cpuidle: using governor menu [ 0.194445] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196450] PCI: Using configuration type 1 for base access [ 0.198124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.208083] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.209015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211088] cryptd: max_cpu_qlen set to 1000 [ 0.215289] ACPI: Added _OSI(Module Device) [ 0.216014] ACPI: Added _OSI(Processor Device) [ 0.218017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224056] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.230555] ACPI: Interpreter enabled [ 0.232053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233014] ACPI: Using IOAPIC for interrupt routing [ 0.235104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.238408] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248628] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.254021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.257086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.261596] acpiphp: Slot [2] registered [ 0.263121] acpiphp: Slot [5] registered [ 0.264151] acpiphp: Slot [6] registered [ 0.265136] acpiphp: Slot [7] registered [ 0.267112] acpiphp: Slot [8] registered [ 0.268112] acpiphp: Slot [9] registered [ 0.269095] acpiphp: Slot [10] registered [ 0.271093] acpiphp: Slot [3] registered [ 0.272084] acpiphp: Slot [4] registered [ 0.273104] acpiphp: Slot [11] registered [ 0.274085] acpiphp: Slot [12] registered [ 0.275119] acpiphp: Slot [13] registered [ 0.277088] acpiphp: Slot [14] registered [ 0.278095] acpiphp: Slot [15] registered [ 0.280103] acpiphp: Slot [16] registered [ 0.281117] acpiphp: Slot [17] registered [ 0.283103] acpiphp: Slot [18] registered [ 0.284147] acpiphp: Slot [19] registered [ 0.285100] acpiphp: Slot [20] registered [ 0.287088] acpiphp: Slot [21] registered [ 0.289137] acpiphp: Slot [22] registered [ 0.291182] acpiphp: Slot [23] registered [ 0.292065] acpiphp: Slot [24] registered [ 0.293093] acpiphp: Slot [25] registered [ 0.295103] acpiphp: Slot [26] registered [ 0.296106] acpiphp: Slot [27] registered [ 0.297109] acpiphp: Slot [28] registered [ 0.299146] acpiphp: Slot [29] registered [ 0.300136] acpiphp: Slot [30] registered [ 0.302104] acpiphp: Slot [31] registered [ 0.303088] PCI host bridge to bus 0000:00 [ 0.305022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319248] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.322056] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326217] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335862] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.341043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351337] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353815] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356211] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360217] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.379016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.383994] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.389212] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.402021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454298] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.519396] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.527016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.533022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.548023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.561384] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.567017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.574020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.601016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.612177] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.621021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.626016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.678565] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.686017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.692015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.705019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.720089] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.722392] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.724266] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.726214] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.727310] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.731134] iommu: Default domain type: Passthrough [ 0.733536] SCSI subsystem initialized [ 0.737156] ACPI: bus type USB registered [ 0.738112] usbcore: registered new interface driver usbfs [ 0.742094] usbcore: registered new interface driver hub [ 0.745140] usbcore: registered new device driver usb [ 0.747250] pps_core: LinuxPPS API ver. 1 registered [ 0.749011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.752062] PTP clock support registered [ 0.753138] EDAC MC: Ver: 3.0.0 [ 0.755146] PCI: Using ACPI for IRQ routing [ 0.756694] NetLabel: Initializing [ 0.758016] NetLabel: domain hash size = 128 [ 0.759011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.762087] NetLabel: unlabeled traffic allowed by default [ 0.765039] vgaarb: loaded [ 0.766252] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.768012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.777000] clocksource: Switched to clocksource kvm-clock [ 0.867648] VFS: Disk quotas dquot_6.6.0 [ 0.868749] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.870409] *** VALIDATE ramfs *** [ 0.871359] *** VALIDATE hugetlbfs *** [ 0.872785] pnp: PnP ACPI init [ 0.874641] pnp: PnP ACPI: found 6 devices [ 0.891190] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.894458] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.896548] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.898589] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.900907] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.903216] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.905946] NET: Registered protocol family 2 [ 0.908331] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.912745] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.916373] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.921823] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.925988] TCP: Hash tables configured (established 65536 bind 65536) [ 0.928571] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.931372] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.933788] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.936682] NET: Registered protocol family 1 [ 0.939198] RPC: Registered named UNIX socket transport module. [ 0.941097] RPC: Registered udp transport module. [ 0.942735] RPC: Registered tcp transport module. [ 0.943893] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.946020] NET: Registered protocol family 44 [ 0.947327] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.949400] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.950996] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.952652] PCI: CLS 0 bytes, default 64 [ 0.953955] Unpacking initramfs... [ 2.348514] debug: unmapping init [mem 0xffff8a96fcc54000-0xffff8a96fffbffff] [ 2.352142] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.354036] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.356646] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.879140] Initialise system trusted keyrings [ 2.881069] Key type blacklist registered [ 2.882975] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.895286] zbud: loaded [ 2.898667] *** VALIDATE nfs *** [ 2.899932] *** VALIDATE nfs4 *** [ 2.902135] pstore: using deflate compression [ 2.905445] Platform Keyring initialized [ 3.016413] NET: Registered protocol family 38 [ 3.017935] Key type asymmetric registered [ 3.019043] Asymmetric key parser 'x509' registered [ 3.020871] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.024150] io scheduler mq-deadline registered [ 3.025781] io scheduler kyber registered [ 3.027420] io scheduler bfq registered [ 3.029354] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.032217] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.034899] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.037840] ACPI: Power Button [PWRF] [ 3.044494] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.051288] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.066643] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.075632] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.097772] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.124637] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.152731] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.158387] Non-volatile memory driver v1.3 [ 3.160409] Linux agpgart interface v0.103 [ 3.194452] virtio_blk virtio1: [vda] 145952 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.197713] vda: detected capacity change from 0 to 74727424 [ 3.214651] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.217049] vdb: detected capacity change from 0 to 1073741824 [ 3.234781] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.237527] vdc: detected capacity change from 0 to 2621440000 [ 3.264189] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.266600] vdd: detected capacity change from 0 to 2621440000 [ 3.283063] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.286643] vde: detected capacity change from 0 to 4294967296 [ 3.302109] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.305163] vdf: detected capacity change from 0 to 4294967296 [ 3.311866] libphy: Fixed MDIO Bus: probed [ 3.316504] usbcore: registered new interface driver usbserial_generic [ 3.319218] usbserial: USB Serial support registered for generic [ 3.321480] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.324442] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.326226] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.328527] mousedev: PS/2 mouse device common for all mice [ 3.331993] rtc_cmos 00:05: RTC can wake from S4 [ 3.335268] rtc_cmos 00:05: registered as rtc0 [ 3.335314] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.336924] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.342412] intel_pstate: CPU model not supported [ 3.343789] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.348107] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.348192] hid: raw HID events driver (C) Jiri Kosina [ 3.352144] usbcore: registered new interface driver usbhid [ 3.353416] usbhid: USB HID core driver [ 3.354411] drop_monitor: Initializing network drop monitor service [ 3.356904] Initializing XFRM netlink socket [ 3.358752] NET: Registered protocol family 10 [ 3.362273] Segment Routing with IPv6 [ 3.364099] NET: Registered protocol family 17 [ 3.366628] mpls_gso: MPLS GSO support [ 3.372183] RAS: Correctable Errors collector initialized. [ 3.374191] AVX version of gcm_enc/dec engaged. [ 3.375657] AES CTR mode by8 optimization enabled [ 3.459692] sched_clock: Marking stable (3459588653, 0)->(4277248169, -817659516) [ 3.462808] registered taskstats version 1 [ 3.464974] Loading compiled-in X.509 certificates [ 3.467279] zswap: loaded using pool lzo/zbud [ 3.498632] Key type big_key registered [ 3.513328] Key type encrypted registered [ 3.514911] ima: No TPM chip found, activating TPM-bypass! [ 3.517299] ima: Allocated hash algorithm: sha1 [ 3.519014] ima: No architecture policies found [ 3.520970] evm: Initialising EVM extended attributes: [ 3.522879] evm: security.selinux [ 3.524178] evm: security.ima [ 3.525531] evm: security.capability [ 3.526885] evm: HMAC attrs: 0x1 [ 3.528663] rtc_cmos 00:05: setting system clock to 2026-08-06 11:47:31 UTC (1786016851) [ 3.532743] debug: unmapping init [mem 0xffffffffbe403000-0xffffffffbe5fffff] [ 3.535061] debug: unmapping init [mem 0xffffffffbd182000-0xffffffffbd458fff] [ 3.542055] Write protecting the kernel read-only data: 28672k [ 3.544903] debug: unmapping init [mem 0xffffffffbb803000-0xffffffffbb9fffff] [ 3.546840] debug: unmapping init [mem 0xffffffffbc114000-0xffffffffbc1fffff] [ 3.579354] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.586534] systemd[1]: Detected virtualization kvm. [ 3.588877] systemd[1]: Detected architecture x86-64. [ 3.591581] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.621444] systemd[1]: No hostname configured. [ 3.623625] systemd[1]: Set hostname to . [ 3.626108] random: systemd: uninitialized urandom read (16 bytes read) [ 3.629205] systemd[1]: Initializing machine ID from random generator. [ 3.690691] random: ln: uninitialized urandom read (6 bytes read) [ 3.785518] random: systemd: uninitialized urandom read (16 bytes read) [ 3.789159] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.794499] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.797705] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.437991] device-mapper: uevent: version 1.0.3 [ 4.440106] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.177091] random: fast init done [ 5.204346] virtio_net virtio0 ens2: renamed from eth0 [ 5.279401] scsi host0: ata_piix [ 5.313373] scsi host1: ata_piix [ 5.314791] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.317206] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.159756] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.035338] random: crng init done [ 10.036120] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.530102] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.678407] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.936339] SELinux: Disabled at runtime. [ 11.994276] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.000542] systemd[1]: Detected virtualization kvm. [ 12.001945] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.503470] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.509219] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.514539] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.519589] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.524227] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.535728] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.546409] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.681483] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.963991] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.307514] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.319023] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.465192] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.475332] EDAC sbridge: Ver: 1.1.2 [ 14.978436] Key type dns_resolver registered [ 15.287781] NFS: Registering the id_resolver key type [ 15.289787] Key type id_resolver registered [ 15.291428] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg138-server login: [ 42.225145] libcfs: loading out-of-tree module taints kernel. [ 42.247911] Key type ._llcrypt registered [ 42.249573] Key type .llcrypt registered [ 42.298620] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 50.654722] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 51.448818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.456886] alg: No test for adler32 (adler32-zlib) [ 52.502700] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 52.850197] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 54.487205] Key type lgssc registered [ 56.032697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.763780] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 117.515180] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 125.427603] hrtimer: interrupt took 1645142 ns [ 131.165192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 131.256542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.628062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 132.662144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 132.755497] Lustre: lustre-MDT0000: new disk, initializing [ 132.834918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.855436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 137.476454] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.192502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.377042] Lustre: 6481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 152.452841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 152.460024] Lustre: Skipped 1 previous similar message [ 152.642682] Lustre: lustre-MDT0001: new disk, initializing [ 152.781392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.831830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 152.854330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.921451] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.431267] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.860569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.089245] Lustre: lustre-OST0000: new disk, initializing [ 176.091401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.098137] Lustre: 8418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.167928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.594484] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.426519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.434388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.535604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.827987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 195.922924] Lustre: lustre-OST0001: new disk, initializing [ 195.927767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 195.931956] Lustre: 9493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.010641] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.251769] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.753665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 201.763260] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 201.864383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.545969] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.417771] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 227.422738] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.495750] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing yml_node [ 237.479767] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 240.215495] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 242.618854] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 244.243117] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 07:51:31 EDT 2026 [ 261.847566] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 263.660838] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 281.056601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 281.067093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.103461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.616354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.618219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.627243] Lustre: Skipped 1 previous similar message [ 283.632238] Lustre: Skipped 1 previous similar message [ 285.919034] Lustre: server umount lustre-MDT0000 complete [ 289.899894] LustreError: 10368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017137 with bad export cookie 4131255036085011239 [ 289.908053] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 289.912301] LustreError: 10368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 293.856315] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.857176] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.859915] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 293.859927] Lustre: Skipped 2 previous similar messages [ 293.875601] Lustre: Skipped 2 previous similar messages [ 293.896364] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 294.883095] LustreError: 8425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.916499] LustreError: 8425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 296.820098] Lustre: server umount lustre-MDT0001 complete [ 301.385052] Lustre: server umount lustre-OST0000 complete [ 305.504567] Lustre: server umount lustre-OST0001 complete [ 311.698753] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 320.686904] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 359.215049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.573537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 359.607369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 359.763393] Lustre: lustre-MDT0000: new disk, initializing [ 359.846605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.864836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 365.134084] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 378.307939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.369867] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 378.481912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 378.484819] Lustre: Skipped 1 previous similar message [ 378.540934] Lustre: lustre-MDT0001: new disk, initializing [ 378.636245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 378.698874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 378.702796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 385.242916] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 394.619318] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 399.183351] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 406.472793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 406.791125] Lustre: lustre-OST0000: new disk, initializing [ 406.796086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 406.802542] Lustre: 17020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 406.905341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.003979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 408.015103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 408.155920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 411.518191] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 420.117341] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 424.140289] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 424.325442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 428.615279] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 428.764716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 433.638888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 433.644976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.655357] LustreError: Skipped 1 previous similar message [ 433.660055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.665315] Lustre: Skipped 1 previous similar message [ 433.682793] Lustre: Skipped 4 previous similar messages [ 436.723619] Lustre: server umount lustre-OST0000 complete [ 440.289449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 440.295353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.305128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.547954] Lustre: server umount lustre-MDT0000 complete [ 448.993872] LustreError: 15063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.280726] LustreError: 15044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017298 with bad export cookie 4131255036085012772 [ 450.290289] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.302038] LustreError: 15044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 450.582644] Lustre: server umount lustre-MDT0001 complete [ 455.038734] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:55:01 (1786017301) [ 457.062939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 458.341748] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:55:05 (1786017305) [ 465.415716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.962170] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.985625] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 466.069735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.216608] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 471.534718] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.552232] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 479.284087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.323633] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 491.025363] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.478739] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 503.388873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.898730] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 516.507630] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 525.283956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.287313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.294928] Lustre: Skipped 1 previous similar message [ 525.307973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.312127] Lustre: Skipped 2 previous similar messages [ 530.738942] Lustre: server umount lustre-OST0000 complete [ 536.037967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.858994] Lustre: server umount lustre-MDT0000 complete [ 541.670457] LustreError: 19432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.696598] LustreError: 19432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 544.410590] LustreError: 19411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017392 with bad export cookie 4131255036085013647 [ 544.419869] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.428694] LustreError: 19411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 544.933561] Lustre: server umount lustre-MDT0001 complete [ 555.691451] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 558.611987] Key type lgssc unregistered [ 558.895749] LNet: 22964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.903018] LNetError: 22964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.922588] LNet: Removed LNI 192.168.201.138@tcp [ 559.527144] Key type .llcrypt unregistered [ 559.529140] Key type ._llcrypt unregistered [ 584.673584] Key type ._llcrypt registered [ 584.676429] Key type .llcrypt registered [ 584.897933] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 587.003258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 587.047031] alg: No test for adler32 (adler32-zlib) [ 588.064368] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 588.468203] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 590.191513] Key type lgssc registered [ 591.159547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.962940] loop: module loaded [ 602.056809] loop0: detected capacity change from 0 to 204800000 [ 602.292561] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.311801] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.617743] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.085314] loop0: detected capacity change from 0 to 204800000 [ 609.321678] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.330825] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.348752] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.043373] loop0: detected capacity change from 0 to 204800000 [ 620.369218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 620.393057] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.720162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 621.779317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 621.836371] Lustre: lustre-MDT0000: new disk, initializing [ 621.919965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.952387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 626.301378] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 635.679215] loop1: detected capacity change from 0 to 204800000 [ 641.211745] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 641.476632] Lustre: lustre-OST07c6: new disk, initializing [ 641.483192] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 641.493800] Lustre: Skipped 1 previous similar message [ 641.500815] Lustre: 26151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 641.595478] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 647.172749] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 649.841784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 649.856849] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 649.909125] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 652.686271] Lustre: 25138:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 652.694570] Lustre: 25138:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 652.699463] Lustre: 25138:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 652.703934] Lustre: 25138:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 652.711549] Lustre: 25138:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 652.718934] Lustre: 25138:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 665.983361] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 665.993516] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 666.001025] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 666.010718] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 666.033681] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 666.042979] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.723105] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 670.744063] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 670.754970] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 670.760053] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.764426] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 670.769549] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.775877] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 670.782530] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.787405] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 670.791863] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.799253] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.804023] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.001658] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 674.010760] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 674.019381] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 674.034569] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.040829] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 674.049749] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.058530] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 674.063443] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.071533] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 674.076132] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 674.084074] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 674.089829] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.680326] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 678.691944] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 678.700386] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 678.707528] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.720175] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 678.746705] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.763709] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 678.776574] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.790470] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 678.799482] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.814852] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 678.826289] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.024432] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 690.038903] Lustre: 26814:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 690.049873] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 690.057656] Lustre: 26814:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 690.061334] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 690.066620] Lustre: 26814:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 690.070989] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 690.079251] Lustre: 26814:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 690.091881] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 690.098653] Lustre: 26814:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 690.106066] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 690.115424] Lustre: 26814:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 736.739320] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.764525] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 741.858531] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 742.745949] Lustre: server umount lustre-OST07c6 complete [ 752.739745] Lustre: server umount lustre-MDT0000 complete [ 768.057264] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 770.270299] Key type lgssc unregistered [ 770.519591] LNet: 28706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.524981] LNetError: 28706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.558188] LNet: Removed LNI 192.168.201.138@tcp [ 772.289684] Key type .llcrypt unregistered [ 772.291844] Key type ._llcrypt unregistered [ 789.883980] Key type ._llcrypt registered [ 789.885912] Key type .llcrypt registered [ 789.984146] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 790.855425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 790.882161] alg: No test for adler32 (adler32-zlib) [ 791.962640] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 792.146044] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 793.743251] Key type lgssc registered [ 794.998338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 803.928578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 803.945632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.455187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.152608] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 815.591155] LustreError: 29977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.620778] LustreError: 29977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 820.452949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.684267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 826.818824] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 836.013979] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 841.498808] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 848.360794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 848.634506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.680456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 855.242408] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 863.083925] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 881.644852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 881.646167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.651802] LustreError: Skipped 1 previous similar message [ 881.665203] Lustre: Skipped 1 previous similar message [ 881.677035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 881.683282] Lustre: Skipped 1 previous similar message [ 884.679173] Lustre: server umount lustre-OST0000 complete [ 891.874418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 891.878446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 891.888524] Lustre: Skipped 1 previous similar message [ 891.899765] Lustre: Skipped 1 previous similar message [ 896.996604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.008299] Lustre: Skipped 1 previous similar message [ 901.092585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 901.099503] Lustre: Skipped 1 previous similar message [ 902.112380] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 902.258535] Lustre: server umount lustre-MDT0000 complete [ 906.055933] LustreError: 29959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017754 with bad export cookie 921541215427133252 [ 906.059193] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 906.065075] LustreError: 29959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 906.336534] Lustre: server umount lustre-MDT0001 complete [ 916.981516] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 919.412319] Key type lgssc unregistered [ 919.704757] LNet: 33730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.710978] LNetError: 33730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.728411] LNet: Removed LNI 192.168.201.138@tcp [ 920.496292] Key type .llcrypt unregistered [ 920.497815] Key type ._llcrypt unregistered [ 943.004837] Key type ._llcrypt registered [ 943.007838] Key type .llcrypt registered [ 943.122125] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 943.970206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.989852] alg: No test for adler32 (adler32-zlib) [ 945.271029] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 945.527398] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 947.191246] Key type lgssc registered [ 948.102345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 955.502463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 955.518461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.902192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.203397] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 966.114602] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 966.138115] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 968.014729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.307622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 972.057439] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 979.304982] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.012330] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.853327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 992.108496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.129484] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1001.341859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1004.757899] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.804263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.815071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.826957] Lustre: Skipped 1 previous similar message [ 1016.847520] Lustre: Skipped 1 previous similar message [ 1021.839826] Lustre: server umount lustre-OST0000 complete [ 1027.056110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.061899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1027.084815] Lustre: Skipped 1 previous similar message [ 1032.171536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.180868] Lustre: Skipped 2 previous similar messages [ 1032.618283] Lustre: server umount lustre-MDT0000 complete [ 1036.260406] LustreError: 35730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1036.281738] LustreError: 35730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1036.395500] LustreError: 37581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786017884 with bad export cookie 6582007687742754621 [ 1036.406122] LustreError: 37581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1036.412230] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1036.807275] Lustre: server umount lustre-MDT0001 complete [ 1050.152194] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1053.620382] Key type lgssc unregistered [ 1053.900860] LNet: 38536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.916673] LNetError: 38536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.944909] LNet: Removed LNI 192.168.201.138@tcp [ 1054.966032] Key type .llcrypt unregistered [ 1054.968930] Key type ._llcrypt unregistered [ 1077.265900] Key type ._llcrypt registered [ 1077.268959] Key type .llcrypt registered [ 1077.367366] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1078.607246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.650939] alg: No test for adler32 (adler32-zlib) [ 1079.806398] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 1080.012712] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1081.663699] Key type lgssc registered [ 1082.961490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1091.247635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1091.287197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.957274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.598073] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1102.312692] LustreError: 39932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.324873] LustreError: 39932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1106.527546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.976855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.360835] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1121.407760] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1127.532542] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1137.424258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.577906] LustreError: 41751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.688633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.793266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1147.010670] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1154.620825] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1161.751272] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 1162.217108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.245368] Lustre: Skipped 1 previous similar message [ 1162.253886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.255733] Lustre: Skipped 1 previous similar message [ 1162.863354] Lustre: server umount lustre-MDT0000 complete [ 1166.720710] LustreError: 39913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018014 with bad export cookie 8689070686825373172 [ 1166.725906] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.730152] LustreError: 39913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1166.835395] LustreError: 41764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.862942] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.38@tcp (stopping) [ 1167.339373] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.357786] Lustre: Skipped 1 previous similar message [ 1171.365269] LustreError: 39929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.366680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1171.397714] LustreError: 39929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1171.420819] Lustre: Skipped 1 previous similar message [ 1176.484707] LustreError: 41764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.493066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1176.504639] LustreError: 41764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1176.519459] Lustre: Skipped 1 previous similar message [ 1181.151178] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1181.600191] Lustre: server umount lustre-MDT0001 complete [ 1196.758219] Lustre: server umount lustre-OST0000 complete [ 1211.050749] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1213.943788] Key type lgssc unregistered [ 1214.255874] LNet: 43789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.260955] LNetError: 43789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.334385] LNet: Removed LNI 192.168.201.138@tcp [ 1216.296171] Key type .llcrypt unregistered [ 1216.298615] Key type ._llcrypt unregistered [ 1243.183878] Key type ._llcrypt registered [ 1243.186815] Key type .llcrypt registered [ 1243.388652] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1244.913207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1244.921068] alg: No test for adler32 (adler32-zlib) [ 1246.146953] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 1246.289410] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1247.975238] Key type lgssc registered [ 1249.263980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1258.915921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1258.947137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.337700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.308498] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1270.765575] LustreError: 45211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.777292] LustreError: 45211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1273.876476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.239125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1279.228911] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1288.314909] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1295.566304] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1303.743332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1304.158294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.204122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1311.142094] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1318.384367] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1325.025657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.030254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.038875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1326.091384] Lustre: lustre-OST0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 1326.111197] Lustre: Skipped 1 previous similar message [ 1330.146776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1335.271402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1335.283799] Lustre: Skipped 3 previous similar messages [ 1339.359816] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1339.535690] Lustre: server umount lustre-OST0000 complete [ 1360.865621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.881362] Lustre: Skipped 1 previous similar message [ 1360.897422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.912191] Lustre: Skipped 2 previous similar messages [ 1363.800307] Lustre: server umount lustre-MDT0000 complete [ 1365.985087] LustreError: 46396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.010823] LustreError: 46396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1367.009881] LustreError: 45939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.028303] LustreError: 45939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1367.758720] LustreError: 45189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018215 with bad export cookie 5064542082905877784 [ 1367.767068] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.772661] LustreError: 45189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1382.367489] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1382.698904] Lustre: server umount lustre-MDT0001 complete [ 1394.286157] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1397.247669] Key type lgssc unregistered [ 1397.633450] LNet: 48933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.640934] LNetError: 48933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.664823] LNet: Removed LNI 192.168.201.138@tcp [ 1398.750712] Key type .llcrypt unregistered [ 1398.753583] Key type ._llcrypt unregistered [ 1420.437636] Key type ._llcrypt registered [ 1420.439313] Key type .llcrypt registered [ 1420.591925] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1421.812076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1421.836583] alg: No test for adler32 (adler32-zlib) [ 1422.916737] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 1423.232296] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1424.936048] Key type lgssc registered [ 1426.141712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1433.550985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1433.578820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.206700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.859188] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1445.359341] LustreError: 50346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.402192] LustreError: 50346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1449.851348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.191709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1454.711855] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1463.660956] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1469.977498] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1476.654516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1476.952630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.090737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1482.953232] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1491.186645] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1499.816475] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1503.369947] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1515.400336] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-06 8h12m41s [ 1517.065141] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1522.145064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1522.153880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.171560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.249768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.256547] Lustre: Skipped 2 previous similar messages [ 1524.778163] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 1524.793084] Lustre: Skipped 3 previous similar messages [ 1525.511577] Lustre: server umount lustre-MDT0000 complete [ 1528.770748] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1529.318917] LustreError: 50346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.342346] LustreError: 50346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1530.864485] LustreError: 51370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.265809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.370958] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1535.635809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.711364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1536.676085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1539.515178] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1542.895612] Lustre: DEBUG MARKER: Wait for df (36693) ... [ 1544.914119] Lustre: DEBUG MARKER: done [ 1551.330748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1551.342602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.354646] Lustre: Skipped 2 previous similar messages [ 1551.360764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1554.521538] Lustre: server umount lustre-OST0000 complete [ 1557.472583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1557.474459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.480404] LustreError: Skipped 1 previous similar message [ 1557.487963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1557.493690] Lustre: Skipped 2 previous similar messages [ 1557.509600] Lustre: Skipped 4 previous similar messages [ 1563.624446] LustreError: 50345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.686318] Lustre: server umount lustre-MDT0000 complete [ 1567.445482] LustreError: 50327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018415 with bad export cookie 3479811694535487819 [ 1567.447994] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1567.455326] LustreError: 50327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1567.724597] Lustre: server umount lustre-MDT0001 complete [ 1579.227949] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1582.537176] Key type lgssc unregistered [ 1582.895524] LNet: 55271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.901479] LNetError: 55271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.916738] LNet: Removed LNI 192.168.201.138@tcp [ 1583.800063] Key type .llcrypt unregistered [ 1583.803056] Key type ._llcrypt unregistered [ 1623.944986] Key type ._llcrypt registered [ 1623.947735] Key type .llcrypt registered [ 1624.024609] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1624.872305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.073558] alg: No test for adler32 (adler32-zlib) [ 1626.248136] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 1626.496934] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1628.191461] Key type lgssc registered [ 1629.590966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1638.711561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1638.741951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1640.027417] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1640.041684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1640.264032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.899618] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1654.881191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.986653] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1655.025278] Lustre: 57487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1655.365318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1661.688540] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1671.977963] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1678.161892] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1685.029939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1685.276478] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1685.562331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.765106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1691.690774] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1699.109045] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1706.426951] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-06 8h15m53s [ 1708.418440] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1715.780566] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1718.211458] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1718.354887] LustreError: 58433:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1738.779306] Lustre: lustre-MDT0000: Client 66652256-a786-4200-9001-a2150d0ee646 (at 192.168.201.38@tcp) reconnecting [ 1758.416624] LustreError: 58433:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1760.374256] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1762.672764] Lustre: DEBUG MARKER: done [ 1770.534466] Lustre: server umount lustre-OST0000 complete [ 1778.145225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1778.148281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.163253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1778.171697] Lustre: Skipped 1 previous similar message [ 1779.236128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1779.240747] Lustre: Skipped 2 previous similar messages [ 1785.324838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.334470] Lustre: Skipped 1 previous similar message [ 1785.424303] Lustre: server umount lustre-MDT0000 complete [ 1788.384601] Lustre: 56805:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786018620/real 1786018620] req@00000000abdef878 x1872776107448192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786018636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1788.394716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.424199] Lustre: 56805:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1789.211575] LustreError: 57478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018637 with bad export cookie 1404586137079799871 [ 1789.232759] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.234822] LustreError: 57478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1789.651371] Lustre: server umount lustre-MDT0001 complete [ 1800.794614] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 1803.474364] Key type lgssc unregistered [ 1803.699523] LNet: 61820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.714084] LNetError: 61820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.773168] LNet: Removed LNI 192.168.201.138@tcp [ 1805.728526] Key type .llcrypt unregistered [ 1805.730700] Key type ._llcrypt unregistered [ 1855.027526] Key type ._llcrypt registered [ 1855.029192] Key type .llcrypt registered [ 1855.090845] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 1856.431251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.454478] alg: No test for adler32 (adler32-zlib) [ 1857.572705] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 1857.800424] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 1859.479349] Key type lgssc registered [ 1860.699383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1864.336533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1864.345840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.715308] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1865.766090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1866.018094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1867.415163] Lustre: Failing over lustre-MDT0000 [ 1868.228159] Lustre: server umount lustre-MDT0000 complete [ 1877.167711] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:18:43 (1786018723) [ 1883.584210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.038122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.759764] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1895.636647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.749297] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1895.796493] Lustre: 64620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1895.912564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1896.083547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1900.584167] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1909.156917] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1915.268651] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1922.065722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1922.194403] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1922.379494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.439562] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1929.376615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1936.720323] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1941.873917] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1950.178589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.182380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.190175] Lustre: Skipped 2 previous similar messages [ 1950.210515] Lustre: Skipped 2 previous similar messages [ 1954.787402] Lustre: server umount lustre-MDT0000 complete [ 1958.612894] LustreError: 64613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018806 with bad export cookie 16789078963048718029 [ 1958.618379] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.627777] LustreError: 64613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1959.026732] Lustre: server umount lustre-MDT0001 complete [ 1962.934085] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1967.693841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1975.885726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.672627] Lustre: 63542:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786018808/real 1786018808] req@00000000d94513c0 x1872776349654912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786018824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1976.704400] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.871143] Lustre: 63544:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786018811/real 1786018811] req@00000000a414c7a4 x1872776349655040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786018827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1981.983817] Lustre: 63544:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786018813/real 1786018813] req@00000000e1e0c64a x1872776349655168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786018829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.905182] LustreError: 63541:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ec5ffe01 x1872776349655936/t0(0) o250->MGC192.168.201.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1984.209934] LustreError: 66468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.226172] LustreError: 66468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1984.307739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.418430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1985.311353] Lustre: 63542:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786018816/real 1786018816] req@000000003938a3d9 x1872776349655424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786018832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1985.317653] LustreError: 66469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.346198] LustreError: 68388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.385166] LustreError: 68388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1989.342167] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 1989.407098] Lustre: 63543:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786018821/real 1786018821] req@00000000bb6d4148 x1872776349655552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786018837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1991.649444] LustreError: 68387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.669030] LustreError: 68387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1996.772301] LustreError: 68389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.790074] LustreError: 68389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1997.668881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.953090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2002.376353] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2003.432237] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2005.500083] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2010.497496] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2017.513570] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2025.965745] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.976066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.981822] Lustre: Skipped 2 previous similar messages [ 2026.006718] Lustre: Skipped 2 previous similar messages [ 2030.925030] Lustre: server umount lustre-MDT0000 complete [ 2031.072387] LustreError: 66469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.098760] LustreError: 66469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2035.108819] LustreError: 65355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018883 with bad export cookie 16789078963048720311 [ 2035.122480] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2035.136755] LustreError: 65355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2035.623848] Lustre: server umount lustre-MDT0001 complete [ 2043.845915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2052.515045] Lustre: 63542:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786018884/real 1786018884] req@00000000ddc4196e x1872776349683712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786018900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2052.564235] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.399661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.840821] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe8feca5d5280ad1a [ 2060.867499] Lustre: MGC192.168.201.138@tcp: Connection restored to 0@lo (at 0@lo) [ 2061.336629] LustreError: 66468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.423447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.493423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2067.158591] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2077.750464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.399877] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2083.710458] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2092.299772] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2098.637835] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2108.915205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2108.924829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.940432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2111.864345] Lustre: server umount lustre-MDT0000 complete [ 2115.490711] LustreError: 64613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786018963 with bad export cookie 16789078963048721690 [ 2115.493472] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2115.500414] LustreError: 64613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2115.552544] LustreError: 71071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.554194] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2115.567252] LustreError: 71071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2115.582477] Lustre: Skipped 3 previous similar messages [ 2116.079966] Lustre: server umount lustre-MDT0001 complete [ 2123.075304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2126.570224] Lustre: DEBUG MARKER: files compared the same [ 2128.713820] Lustre: server umount lustre-OST0000 complete [ 2143.232452] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2145.714938] Key type lgssc unregistered [ 2145.985796] LNet: 74347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.990331] LNetError: 74347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.010639] LNet: Removed LNI 192.168.201.138@tcp [ 2146.725241] Key type .llcrypt unregistered [ 2146.727069] Key type ._llcrypt unregistered [ 2171.397511] Key type ._llcrypt registered [ 2171.400328] Key type .llcrypt registered [ 2171.505277] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2173.359986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2173.389257] alg: No test for adler32 (adler32-zlib) [ 2174.536336] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 2174.761891] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2176.423643] Key type lgssc registered [ 2177.815081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2186.642395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2186.665185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.189787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.024529] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug +malloc all [ 2198.507471] LustreError: 75757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.537877] LustreError: 75757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2203.362799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.645422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.697145] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug +malloc all [ 2218.320958] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2224.759536] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2232.857088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2233.160653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.240281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2239.553337] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug +malloc all [ 2247.016436] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2258.912028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.915401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2258.926188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2258.930213] Lustre: Skipped 1 previous similar message [ 2264.174988] LustreError: 77581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.346146] Lustre: server umount lustre-OST0000 complete [ 2269.158571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.184482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.187257] Lustre: Skipped 1 previous similar message [ 2274.273938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.283466] Lustre: Skipped 1 previous similar message [ 2274.748412] Lustre: server umount lustre-MDT0000 complete [ 2278.859230] LustreError: 75739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786019126 with bad export cookie 15317874326219654284 [ 2278.866760] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2278.872919] LustreError: 75739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2279.317634] Lustre: server umount lustre-MDT0001 complete [ 2293.276751] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2296.524690] Key type lgssc unregistered [ 2296.862575] LNet: 79294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.872679] LNetError: 79294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.887757] LNet: Removed LNI 192.168.201.138@tcp [ 2298.066501] Key type .llcrypt unregistered [ 2298.067840] Key type ._llcrypt unregistered [ 2325.778602] Key type ._llcrypt registered [ 2325.788027] Key type .llcrypt registered [ 2325.935600] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2327.658238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2327.718643] alg: No test for adler32 (adler32-zlib) [ 2328.757551] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 2328.986125] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2330.649470] Key type lgssc registered [ 2332.523282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2342.130270] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2342.158552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2357.791449] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2362.911649] LustreError: 80686:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.138@tcp: failed processing log, type 4: rc = -110 [ 2389.664018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.773120] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2405.217602] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2414.040397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.274512] Lustre: *** cfs_fail_loc=706, val=0*** [ 2414.724159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.803663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2419.743502] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2420.198448] LustreError: 80713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.220905] LustreError: 80713:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2424.297246] LustreError: 81772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.320982] LustreError: 81772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2428.727394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.115073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2434.109817] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2442.543951] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2449.809520] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2455.014570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.034791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.077104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.130383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.132029] Lustre: Skipped 1 previous similar message [ 2465.251365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.260017] Lustre: Skipped 1 previous similar message [ 2466.271524] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2466.389173] Lustre: server umount lustre-OST0000 complete [ 2474.977101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2474.982595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.990707] Lustre: Skipped 1 previous similar message [ 2474.994449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.997107] Lustre: Skipped 1 previous similar message [ 2477.868581] Lustre: server umount lustre-MDT0000 complete [ 2480.609675] LustreError: 81777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.624495] LustreError: 81777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2481.819052] LustreError: 80692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786019329 with bad export cookie 2654435420632967481 [ 2481.826926] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2481.829737] LustreError: 80692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2482.091840] Lustre: server umount lustre-MDT0001 complete [ 2493.629162] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2496.600423] Key type lgssc unregistered [ 2496.951372] LNet: 84304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.962748] LNetError: 84304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.983091] LNet: Removed LNI 192.168.201.138@tcp [ 2498.003893] Key type .llcrypt unregistered [ 2498.006985] Key type ._llcrypt unregistered [ 2522.441841] Key type ._llcrypt registered [ 2522.443525] Key type .llcrypt registered [ 2522.499585] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2523.415601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2523.537779] alg: No test for adler32 (adler32-zlib) [ 2524.591715] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 2524.874129] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2526.641489] Key type lgssc registered [ 2527.759630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2537.223126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2537.241320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.502994] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2552.377301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.143598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2557.928056] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2558.436278] LustreError: 86409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.455516] LustreError: 86409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2562.528752] LustreError: 86412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.648485] LustreError: 86413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.678504] LustreError: 86413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2567.963776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2568.431134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.794566] LustreError: 86412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.826117] LustreError: 86412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2575.106312] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2577.890258] LustreError: 86413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.949762] LustreError: 86413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2586.330592] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2586.341200] Lustre: MGS: Client af64079c-257c-47fb-a4d9-442c67726c93 (at 0@lo) reconnecting [ 2586.373108] LustreError: 86423:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2586.714253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.806992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2592.832161] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2612.204595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2612.213061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.221832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2612.226404] Lustre: Skipped 1 previous similar message [ 2612.246569] Lustre: Skipped 1 previous similar message [ 2616.437723] Lustre: server umount lustre-OST0000 complete [ 2622.434418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.438069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2622.442520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2622.452817] Lustre: Skipped 1 previous similar message [ 2627.555472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2627.570519] Lustre: Skipped 2 previous similar messages [ 2628.591251] Lustre: server umount lustre-MDT0000 complete [ 2632.690641] LustreError: 87437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.711348] LustreError: 87437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2633.327963] Lustre: server umount lustre-MDT0001 complete [ 2638.731286] Lustre: server umount MGS complete [ 2653.590480] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:31:40 (1786019500) [ 2686.128986] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 2695.527893] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2746.153275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.566719] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2762.965652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.185951] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2763.208506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2763.252685] Lustre: lustre-MDT0001: new disk, initializing [ 2763.307906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2763.336949] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2763.339696] Lustre: 93602:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2763.350912] LustreError: 93602:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2769.027681] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2781.353882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.605415] Lustre: lustre-OST0000: new disk, initializing [ 2781.609847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2781.618733] Lustre: 94370:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2781.678732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.072861] Lustre: 94672:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f9d088d5 x1872777049057664/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786019655 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2791.204988] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2802.537934] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2810.444719] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.465554] Lustre: MGS: Client 429b0f46-2113-48b0-ba54-5f5b3f456754 (at 0@lo) reconnecting [ 2810.490555] LustreError: 93588:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2810.546252] Lustre: 92870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2810.601502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2810.763955] Lustre: lustre-MDT0000: new disk, initializing [ 2811.038232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.089343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2811.133549] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2812.192776] Lustre: 95441:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000c631ff67 x1872777049065216/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786019676 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2817.315567] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2817.441975] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2821.319079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2821.332540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2821.529844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2844.645593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.667645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.671441] Lustre: Skipped 1 previous similar message [ 2844.684470] Lustre: Skipped 1 previous similar message [ 2849.566366] Lustre: server umount lustre-OST0000 complete [ 2854.880607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.895647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.927566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.940929] Lustre: Skipped 1 previous similar message [ 2864.115132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.136518] Lustre: Skipped 3 previous similar messages [ 2868.191142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2868.335052] Lustre: server umount lustre-MDT0000 complete [ 2869.217551] LustreError: 93573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.249204] LustreError: 93573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2879.159385] Lustre: server umount lustre-MDT0001 complete [ 2884.141658] Lustre: server umount MGS complete [ 2900.495383] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:35:46 (1786019746) [ 2927.483799] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 2932.185790] Key type lgssc unregistered [ 2932.634988] LNet: 97944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2932.646498] LNetError: 97944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2932.667366] LNet: Removed LNI 192.168.201.138@tcp [ 2933.806435] Key type .llcrypt unregistered [ 2933.811521] Key type ._llcrypt unregistered [ 2952.674140] Key type ._llcrypt registered [ 2952.679181] Key type .llcrypt registered [ 2952.789062] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 2953.948372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2953.988820] alg: No test for adler32 (adler32-zlib) [ 2955.281709] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 2955.588948] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 2957.304320] Key type lgssc registered [ 2958.563377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2967.197263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2967.231487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.693982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.204607] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2978.283134] LustreError: 99167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.303570] LustreError: 99167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2982.715367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.258285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.546127] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 2996.551568] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3002.859700] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3011.001657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3011.428073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.508286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3018.676378] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3027.703537] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3039.202055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3039.213090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.235473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3039.247198] Lustre: Skipped 1 previous similar message [ 3042.840283] Lustre: server umount lustre-OST0000 complete [ 3049.447381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.455125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3049.476123] Lustre: Skipped 2 previous similar messages [ 3049.485813] Lustre: Skipped 1 previous similar message [ 3053.493886] Lustre: server umount lustre-MDT0000 complete [ 3054.562021] LustreError: 99164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.576426] LustreError: 99164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3057.190951] LustreError: 99148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786019905 with bad export cookie 12709487026538441509 [ 3057.202333] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3057.212575] LustreError: 99148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3057.649346] Lustre: server umount lustre-MDT0001 complete [ 3068.212335] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3071.239564] Key type lgssc unregistered [ 3071.659055] LNet: 102701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3071.670413] LNetError: 102701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3072.742944] LNet: Removed LNI 192.168.201.138@tcp [ 3073.806886] Key type .llcrypt unregistered [ 3073.811790] Key type ._llcrypt unregistered [ 3090.607589] Key type ._llcrypt registered [ 3090.609826] Key type .llcrypt registered [ 3090.746907] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3091.869388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3092.043272] alg: No test for adler32 (adler32-zlib) [ 3093.135189] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 3093.461651] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3095.111203] Key type lgssc registered [ 3096.822513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3104.579677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3104.597590] LustreError: 103781:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3104.681414] LustreError: 103785:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3104.715834] LustreError: 103781:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4897 [ 3104.735313] LustreError: 103781:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3104.762587] LustreError: 103781:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3104.763949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.896176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.024840] LustreError: 103823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3117.035820] LustreError: 103823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3117.779243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.394859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3126.563217] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3132.331610] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing lsmod [ 3136.023168] LustreError: 104992:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3136.030074] LustreError: 104993:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3136.034199] LustreError: 104992:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3136.043178] LustreError: 104992:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3136.047491] LustreError: 104992:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3136.063282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3136.542447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.600893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.605634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.607643] Lustre: Skipped 1 previous similar message [ 3143.651925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.659308] Lustre: Skipped 2 previous similar messages [ 3146.307956] Lustre: server umount lustre-MDT0000 complete [ 3148.791913] LustreError: 103818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.827197] LustreError: 103818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3150.286473] LustreError: 103801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786019998 with bad export cookie 17799477124816281394 [ 3150.293436] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.306304] LustreError: 103801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3150.321608] Lustre: 103814:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3151.967698] LustreError: 104255:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002a5ea550 x1872777645675136/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3152.150090] Lustre: server umount lustre-MDT0001 complete [ 3167.299690] Lustre: server umount lustre-OST0000 complete [ 3177.174829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.036136] LustreError: 106057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3178.194980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.277591] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3183.599542] LustreError: 106058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3188.706534] LustreError: 106057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.678111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.258234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3198.223700] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3205.324747] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3210.935666] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3220.272884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3221.080445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3223.197937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3230.388135] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3239.685778] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3249.121510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3249.135469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.139247] LustreError: Skipped 1 previous similar message [ 3249.153249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.164469] Lustre: Skipped 2 previous similar messages [ 3249.184961] Lustre: Skipped 1 previous similar message [ 3252.809430] Lustre: server umount lustre-OST0000 complete [ 3259.360839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.369732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.370729] Lustre: Skipped 1 previous similar message [ 3259.397910] Lustre: Skipped 1 previous similar message [ 3263.726761] Lustre: server umount lustre-MDT0000 complete [ 3264.482498] LustreError: 107053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.489359] LustreError: 107053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3266.922767] LustreError: 106039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786020114 with bad export cookie 17799477124816282094 [ 3266.924788] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3266.946956] LustreError: 106039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3267.272716] Lustre: server umount lustre-MDT0001 complete [ 3279.987548] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3282.920481] Key type lgssc unregistered [ 3283.341917] LNet: 109593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3283.349556] LNetError: 109593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3283.365385] LNet: Removed LNI 192.168.201.138@tcp [ 3284.441491] Key type .llcrypt unregistered [ 3284.443390] Key type ._llcrypt unregistered [ 3312.032615] Key type ._llcrypt registered [ 3312.036701] Key type .llcrypt registered [ 3312.179409] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3313.862107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3313.913463] alg: No test for adler32 (adler32-zlib) [ 3315.089783] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 3315.285037] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3316.935166] Key type lgssc registered [ 3318.408735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3327.499786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3327.543536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.114435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.072920] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3339.237592] LustreError: 111010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.271551] LustreError: 111010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3344.359793] LustreError: 111011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.733296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.273473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.592736] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3360.724645] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3366.701283] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3374.205497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3374.735223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.029317] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3390.265559] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3399.252566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3407.272181] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3411.611393] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3418.599112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3418.616567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.636711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.616540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.640465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.643427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.766177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.777288] Lustre: Skipped 2 previous similar messages [ 3429.876575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.882175] Lustre: Skipped 1 previous similar message [ 3431.393333] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3431.803952] Lustre: server umount lustre-OST0000 complete [ 3439.584608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3439.596404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.611170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.121759] Lustre: server umount lustre-MDT0000 complete [ 3445.225833] LustreError: 111010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.237340] LustreError: 111010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3449.353835] LustreError: 110990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786020297 with bad export cookie 13842571298117831001 [ 3449.365461] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3450.100377] Lustre: server umount lustre-MDT0001 complete [ 3465.630235] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3469.357138] Key type lgssc unregistered [ 3469.687684] LNet: 114679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3469.694477] LNetError: 114679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3469.713246] LNet: Removed LNI 192.168.201.138@tcp [ 3471.202626] Key type .llcrypt unregistered [ 3471.207702] Key type ._llcrypt unregistered [ 3494.940927] Key type ._llcrypt registered [ 3494.944501] Key type .llcrypt registered [ 3495.189387] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3498.100590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3498.288232] alg: No test for adler32 (adler32-zlib) [ 3499.844922] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 3500.482711] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3502.383283] Key type lgssc registered [ 3503.771928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3512.423508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3512.450415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.062956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.363409] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3523.562586] LustreError: 115903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.590015] LustreError: 115903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3528.674971] LustreError: 115904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.823328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.276023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3533.803964] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3541.670319] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3546.505819] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3555.219124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3555.709281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.798400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3563.975280] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3572.975345] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3584.485301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3584.490708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3584.505407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3584.516342] Lustre: Skipped 1 previous similar message [ 3589.604563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.612601] Lustre: Skipped 2 previous similar messages [ 3594.723205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.255204] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3596.507892] Lustre: server umount lustre-OST0000 complete [ 3604.961390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.978085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.978893] Lustre: Skipped 1 previous similar message [ 3605.008363] Lustre: Skipped 2 previous similar messages [ 3608.155785] Lustre: server umount lustre-MDT0000 complete [ 3610.081330] LustreError: 115899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.116755] LustreError: 115899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3612.938585] LustreError: 115883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786020460 with bad export cookie 9385359118717346039 [ 3612.959495] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3612.969600] LustreError: 115883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3613.534154] Lustre: server umount lustre-MDT0001 complete [ 3627.669432] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3631.055587] Key type lgssc unregistered [ 3631.405780] LNet: 119439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3631.414376] LNetError: 119439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3631.432042] LNet: Removed LNI 192.168.201.138@tcp [ 3632.498656] Key type .llcrypt unregistered [ 3632.502694] Key type ._llcrypt unregistered [ 3661.108678] Key type ._llcrypt registered [ 3661.111320] Key type .llcrypt registered [ 3661.261257] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3662.870286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3663.009804] alg: No test for adler32 (adler32-zlib) [ 3664.049749] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 3664.299768] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3665.992074] Key type lgssc registered [ 3667.451971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3676.834437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3676.862562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.602807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.971969] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3688.945139] LustreError: 120872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.981114] LustreError: 120872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3694.056833] LustreError: 120873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.082148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.550665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3699.711455] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3708.042628] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3713.403470] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3721.393048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3722.084336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.163918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3730.799960] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3742.238909] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3750.369592] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3754.629869] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3754.633594] Lustre: Skipped 2 previous similar messages [ 3759.779892] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3759.808740] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3759.921299] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3759.927568] Lustre: Skipped 2 previous similar messages [ 3770.426040] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3770.440596] Lustre: Skipped 1 previous similar message [ 3775.201994] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.38@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3775.211853] Lustre: Skipped 2 previous similar messages [ 3776.440777] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.38@tcp 192.168.0.[2,10]@tcp [ 3786.090715] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.38/24@tcp in log lustre-MDT0000 [ 3786.097231] Lustre: Skipped 2 previous similar messages [ 3793.013217] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.38/24@tcp [ 3793.019763] Lustre: Skipped 1 previous similar message [ 3801.057719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3801.067603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.087464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.732900] Lustre: server umount lustre-OST0000 complete [ 3807.203775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3807.214212] LustreError: Skipped 5 previous similar messages [ 3807.218562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.234473] Lustre: Skipped 1 previous similar message [ 3807.239753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3807.242160] Lustre: Skipped 1 previous similar message [ 3811.299306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.302116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.709148] Lustre: server umount lustre-MDT0000 complete [ 3816.422900] LustreError: 120872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.446689] LustreError: 120872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3816.911532] LustreError: 120852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786020664 with bad export cookie 8239482229952766597 [ 3816.921472] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.929208] LustreError: 120852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3817.191773] Lustre: server umount lustre-MDT0001 complete [ 3830.921478] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 3834.844682] Key type lgssc unregistered [ 3835.034789] LNet: 125873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3835.040809] LNetError: 125873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3835.050366] LNet: Removed LNI 192.168.201.138@tcp [ 3835.732143] Key type .llcrypt unregistered [ 3835.733578] Key type ._llcrypt unregistered [ 3872.119612] Key type ._llcrypt registered [ 3872.121996] Key type .llcrypt registered [ 3872.203359] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 3873.331609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3873.441106] alg: No test for adler32 (adler32-zlib) [ 3874.514726] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 3874.737860] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 3876.415183] Key type lgssc registered [ 3877.821483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3886.082519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3886.104835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.712934] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3887.743204] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.38/24@tcp [ 3887.774846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.674692] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3897.327688] LustreError: 127439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3897.359939] LustreError: 127439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3902.433252] LustreError: 127438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.196312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.570456] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3903.604694] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.38/24@tcp [ 3903.653587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3909.216685] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3918.352738] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3926.196609] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3934.698720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3935.127623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.783488] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 3953.822347] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3957.741199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3973.089192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.094627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.097667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.109482] Lustre: Skipped 1 previous similar message [ 3976.598520] Lustre: server umount lustre-OST0000 complete [ 3983.331969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.343029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3983.354015] Lustre: Skipped 2 previous similar messages [ 3987.003737] Lustre: server umount lustre-MDT0000 complete [ 3988.462256] LustreError: 127439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.501703] LustreError: 127439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3991.414917] LustreError: 127420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786020839 with bad export cookie 13844961513894272677 [ 3991.422980] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3991.424590] LustreError: 127420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3991.828544] Lustre: server umount lustre-MDT0001 complete [ 4008.238929] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4014.225712] Key type lgssc unregistered [ 4014.575700] LNet: 131051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4014.581845] LNetError: 131051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4014.599281] LNet: Removed LNI 192.168.201.138@tcp [ 4016.139411] Key type .llcrypt unregistered [ 4016.140733] Key type ._llcrypt unregistered [ 4052.721645] Key type ._llcrypt registered [ 4052.724324] Key type .llcrypt registered [ 4052.905375] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4069.393639] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4071.037454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4071.067058] alg: No test for adler32 (adler32-zlib) [ 4072.239628] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 4072.474763] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4074.231921] Key type lgssc registered [ 4075.484715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4119.929526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4119.955824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.333647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4121.416247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4121.544629] Lustre: lustre-MDT0000: new disk, initializing [ 4121.711417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4121.749854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4128.463408] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4140.080830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.199243] Lustre: 134454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4140.252667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4140.256364] Lustre: Skipped 1 previous similar message [ 4140.382918] Lustre: lustre-MDT0001: new disk, initializing [ 4140.466981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4140.504237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4140.514410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4145.257199] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4157.929561] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4166.975114] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4176.079624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4176.383970] Lustre: lustre-OST0000: new disk, initializing [ 4176.389342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4176.400709] Lustre: 136420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4176.511479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.731089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4183.739404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4183.929680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4185.834456] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4198.592284] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4204.805453] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4205.043445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4211.189926] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4211.604197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4214.754121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.755172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4214.761765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4219.877023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4219.883237] Lustre: Skipped 2 previous similar messages [ 4224.992573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.003913] Lustre: Skipped 1 previous similar message [ 4229.088305] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4229.220408] Lustre: server umount lustre-OST0000 complete [ 4235.235037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.244550] Lustre: Skipped 1 previous similar message [ 4235.249674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.253571] Lustre: Skipped 1 previous similar message [ 4239.995434] Lustre: server umount lustre-MDT0000 complete [ 4245.478090] LustreError: 136622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.521430] LustreError: 136622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4245.718279] LustreError: 134446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786021093 with bad export cookie 11998620530933646664 [ 4245.725228] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4245.737530] LustreError: 134446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4246.830387] Lustre: server umount lustre-MDT0001 complete [ 4259.353675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.350342] LustreError: 138642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.542759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.969188] LustreError: 138643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.014747] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4270.049650] LustreError: 138642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.168339] LustreError: 138643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.806770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4277.562532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4283.461332] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4291.621314] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4297.323244] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4305.806362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4306.110943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.857587] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4322.438566] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4329.013928] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4331.550576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4337.028749] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4337.334406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4345.414915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4345.579436] Lustre: lustre-OST0001: new disk, initializing [ 4345.584590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4345.592958] Lustre: 142031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4345.710572] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4352.739679] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4355.212980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4355.225268] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4355.292944] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4366.751614] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4366.973544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4374.720077] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4375.309576] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4385.443078] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97a247488000.ost_server_uuid 50 [ 4388.116525] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97a247488000.ost_server_uuid in IDLE state after 0 sec [ 4401.635203] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4401.635994] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.637708] LustreError: Skipped 1 previous similar message [ 4401.638183] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4401.662695] Lustre: Skipped 2 previous similar messages [ 4411.893236] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4411.902539] Lustre: Skipped 3 previous similar messages [ 4414.943984] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4415.191755] Lustre: server umount lustre-OST0001 complete [ 4417.020124] LustreError: 143443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.036150] LustreError: 143443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4420.065482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4420.083567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.981527] Lustre: server umount lustre-OST0000 complete [ 4431.328609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4431.342882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.353633] Lustre: Skipped 1 previous similar message [ 4431.358280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4431.362272] Lustre: Skipped 3 previous similar messages [ 4436.748641] Lustre: server umount lustre-MDT0000 complete [ 4437.481896] LustreError: 139370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4437.506039] LustreError: 139370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4440.547884] LustreError: 138626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786021288 with bad export cookie 11998620530933647546 [ 4440.549299] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4440.564694] LustreError: 138626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4441.049651] Lustre: server umount lustre-MDT0001 complete [ 4466.017581] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4469.849732] Key type lgssc unregistered [ 4470.212745] LNet: 145068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4470.218224] LNetError: 145068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4471.299740] LNet: Removed LNI 192.168.201.138@tcp [ 4472.316301] Key type .llcrypt unregistered [ 4472.322844] Key type ._llcrypt unregistered [ 4523.086269] Key type ._llcrypt registered [ 4523.087789] Key type .llcrypt registered [ 4523.162957] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4541.535219] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4542.731209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4542.746877] alg: No test for adler32 (adler32-zlib) [ 4543.838490] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 4544.093701] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4545.784083] Key type lgssc registered [ 4546.825745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4590.614899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4590.638921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4591.973665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4592.021492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4592.133819] Lustre: lustre-MDT0000: new disk, initializing [ 4592.235859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4592.260283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4597.726570] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4609.043749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4609.208399] Lustre: 149241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4609.335510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4609.342047] Lustre: Skipped 1 previous similar message [ 4609.577751] Lustre: lustre-MDT0001: new disk, initializing [ 4609.840604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4609.946184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4609.964209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4615.667766] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4629.896638] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4638.113748] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4646.112113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4646.387744] Lustre: lustre-OST0000: new disk, initializing [ 4646.390443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4646.394080] Lustre: 151214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4646.451191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.355630] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4656.270282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4656.292350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4656.473040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4664.440097] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4670.980462] Lustre: Failing over lustre-OST0000 [ 4671.136621] Lustre: server umount lustre-OST0000 complete [ 4687.314297] Lustre: 147467:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021519/real 1786021519] req@0000000096c03d95 x1872779167173504/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786021535 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4687.374422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.834902] Lustre: 147468:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021519/real 1786021519] req@00000000970036d0 x1872779167172608/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786021535 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4687.891263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.535877] Lustre: 147467:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021525/real 1786021525] req@00000000a3de5a6a x1872779167174528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786021541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4693.569693] Lustre: 147467:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4693.680608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.447677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4694.495607] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4697.517199] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4698.239522] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4698.252518] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4699.303547] Lustre: 147469:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021530/real 1786021530] req@000000001fb1043a x1872779167175680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786021546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4699.348206] Lustre: 147469:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4702.853836] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4703.455710] Lustre: 147469:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021535/real 1786021535] req@000000008063507b x1872779167176832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786021551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4703.477062] Lustre: 147469:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4707.053320] Lustre: Failing over lustre-MDT0000 [ 4707.297630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4707.302122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.489117] Lustre: server umount lustre-MDT0000 complete [ 4712.930674] LustreError: 149254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.959993] LustreError: 149254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4713.471534] LustreError: 149249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4718.050582] LustreError: 149250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4718.064534] LustreError: 149250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4723.171905] LustreError: 149253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4723.227374] LustreError: 149253:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4728.287929] Lustre: 147467:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021560/real 1786021560] req@000000007acb6693 x1872779167188224/t0(0) o400->MGC192.168.201.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786021576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4728.330641] LustreError: 149249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.340639] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4728.402436] LustreError: 149249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4729.237481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.528909] LustreError: 149254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.561101] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1e31d81ab81fd254 [ 4738.571345] LustreError: 149254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4738.607838] Lustre: MGC192.168.201.138@tcp: Connection restored to 0@lo (at 0@lo) [ 4738.614029] Lustre: Skipped 1 previous similar message [ 4738.933769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4739.020234] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4739.049299] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4744.235153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4744.301487] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4744.411857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4745.620034] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4757.938099] Lustre: server umount lustre-OST0000 complete [ 4764.649307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4764.652768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4764.661122] Lustre: Skipped 3 previous similar messages [ 4769.762111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4769.768076] Lustre: Skipped 2 previous similar messages [ 4771.880787] Lustre: server umount lustre-MDT0000 complete [ 4774.890925] LustreError: 149248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.915771] LustreError: 149248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4775.199420] Lustre: 147469:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786021607/real 1786021607] req@000000003c86a7f8 x1872779167207040/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786021623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4775.224914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.058647] LustreError: 149234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786021624 with bad export cookie 2175757704266502740 [ 4776.074274] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4776.625549] Lustre: server umount lustre-MDT0001 complete [ 4790.685664] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 4793.783469] Key type lgssc unregistered [ 4794.100719] LNet: 154870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4794.115490] LNetError: 154870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4794.137629] LNet: Removed LNI 192.168.201.138@tcp [ 4795.172590] Key type .llcrypt unregistered [ 4795.177658] Key type ._llcrypt unregistered [ 4837.507645] Key type ._llcrypt registered [ 4837.511710] Key type .llcrypt registered [ 4837.614303] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4839.471845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4839.497221] alg: No test for adler32 (adler32-zlib) [ 4840.718692] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 4841.040095] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 4842.775322] Key type lgssc registered [ 4844.088862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4852.404608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4852.415466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.910464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4858.745551] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4863.458988] LustreError: 156391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.470857] LustreError: 156391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4868.580710] LustreError: 156392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4868.637922] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4872.211220] LustreError: 156420:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4872.226160] LustreError: 156420:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4872.233260] LustreError: 156420:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4872.635186] Lustre: server umount lustre-MDT0000 complete [ 4887.098964] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 4897.431582] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 4943.336706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.646442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4943.688435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4943.761223] Lustre: lustre-MDT0000: new disk, initializing [ 4943.832140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4943.860756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4949.322882] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4961.927600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.103247] Lustre: 159725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4962.170798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4962.172870] Lustre: Skipped 1 previous similar message [ 4962.250537] Lustre: lustre-MDT0001: new disk, initializing [ 4962.348128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4962.387903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4962.396806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4967.458890] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 4979.393212] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4986.601282] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4996.136860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4996.427846] Lustre: lustre-OST0000: new disk, initializing [ 4996.430959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4996.442744] Lustre: 161694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4996.532608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5004.327032] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5005.460618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5005.475772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5005.705285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5015.323465] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5019.952422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5021.832872] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:11:08 (1786021868) [ 5282.277036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.278232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5282.288937] Lustre: Skipped 1 previous similar message [ 5282.298164] Lustre: Skipped 2 previous similar messages [ 5287.392967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.398625] Lustre: Skipped 2 previous similar messages [ 5287.724818] Lustre: server umount lustre-MDT0000 complete [ 5291.325285] LustreError: 159715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786022139 with bad export cookie 3574074672337138187 [ 5291.332398] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.339583] LustreError: 159715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5292.516150] LustreError: 159731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5292.517446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.542226] Lustre: Skipped 1 previous similar message [ 5292.559927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5296.546071] LustreError: 163463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.546244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5297.941535] Lustre: server umount lustre-MDT0001 complete [ 5307.873133] Lustre: server umount lustre-OST0000 complete [ 5317.161259] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5328.966790] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5373.489232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5373.791925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5373.813623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5373.881432] Lustre: lustre-MDT0000: new disk, initializing [ 5373.997772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5374.015853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5379.034641] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5388.572054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.734119] Lustre: 166358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5388.831106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5388.849117] Lustre: Skipped 1 previous similar message [ 5388.958821] Lustre: lustre-MDT0001: new disk, initializing [ 5389.078624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5389.108209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5389.125076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5393.684966] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5403.602248] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5409.448683] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5416.081842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5416.367662] Lustre: lustre-OST0000: new disk, initializing [ 5416.375723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5416.382280] Lustre: 168323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5416.461065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5422.086907] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5424.211542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5424.220093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5424.302470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5431.201253] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5435.934876] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5436.158833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5441.247095] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5441.466230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5445.089443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.091147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.104541] LustreError: Skipped 1 previous similar message [ 5445.131133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5445.152997] Lustre: Skipped 1 previous similar message [ 5449.670553] Lustre: server umount lustre-OST0000 complete [ 5455.328683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.338397] Lustre: Skipped 1 previous similar message [ 5455.342363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5455.352104] Lustre: Skipped 1 previous similar message [ 5459.732754] Lustre: server umount lustre-MDT0000 complete [ 5460.449288] LustreError: 166369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5460.486108] LustreError: 166369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5464.014824] LustreError: 166348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786022311 with bad export cookie 3574074672337297017 [ 5464.019336] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.028608] LustreError: 166348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5464.313617] Lustre: server umount lustre-MDT0001 complete [ 5472.827174] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:18:39 (1786022319) [ 5488.645183] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5499.527927] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5545.556316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5545.867848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5545.895909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5546.009422] Lustre: lustre-MDT0000: new disk, initializing [ 5546.122170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5546.137202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5551.617702] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5563.855787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.985987] Lustre: 173082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5564.278809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5569.512328] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5580.708269] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5586.953322] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5593.527332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5593.920571] Lustre: lustre-OST0000: new disk, initializing [ 5593.926934] Lustre: Skipped 1 previous similar message [ 5593.934547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5593.941974] Lustre: Skipped 2 previous similar messages [ 5593.949217] Lustre: 175059:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5601.350519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5601.368946] Lustre: Skipped 1 previous similar message [ 5601.374680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5601.517814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5602.066474] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5616.521580] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5627.359962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5627.363816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.384200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5627.402846] Lustre: Skipped 2 previous similar messages [ 5632.482647] Lustre: server umount lustre-OST0000 complete [ 5637.606154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.623169] Lustre: Skipped 1 previous similar message [ 5642.723724] LustreError: 173092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5643.071809] Lustre: server umount lustre-MDT0000 complete [ 5647.437277] LustreError: 173073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786022495 with bad export cookie 3574074672337297899 [ 5647.449815] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5647.451914] LustreError: 173073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5647.758538] Lustre: server umount lustre-MDT0001 complete [ 5656.987954] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:21:43 (1786022503) [ 5675.689581] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 5689.190683] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5733.589349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.062828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5734.160648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5734.337165] Lustre: lustre-MDT0000: new disk, initializing [ 5734.545982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.556912] Lustre: Skipped 2 previous similar messages [ 5734.576537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5741.541073] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5754.277410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.458885] Lustre: 179397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5754.860577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5760.197582] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5770.910810] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5778.001311] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5785.889552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5786.296560] Lustre: 181373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5792.365781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5792.501361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5793.184582] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5804.267265] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5813.220259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.222937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5813.236013] LustreError: Skipped 1 previous similar message [ 5813.239452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.246906] Lustre: Skipped 4 previous similar messages [ 5818.343729] LustreError: 181352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.369022] LustreError: 181352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5818.553692] Lustre: server umount lustre-OST0000 complete [ 5829.732414] Lustre: server umount lustre-MDT0000 complete [ 5833.817271] LustreError: 179388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786022681 with bad export cookie 3574074672337298984 [ 5833.821371] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5833.838555] LustreError: 179388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5834.243635] Lustre: server umount lustre-MDT0001 complete [ 5847.239791] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 5850.384344] Key type lgssc unregistered [ 5850.698213] LNet: 183229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5850.707084] LNetError: 183229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5850.717826] LNet: Removed LNI 192.168.201.138@tcp [ 5851.633645] Key type .llcrypt unregistered [ 5851.635713] Key type ._llcrypt unregistered [ 5879.758561] Key type ._llcrypt registered [ 5879.761209] Key type .llcrypt registered [ 5879.873609] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 5881.143853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5881.187112] alg: No test for adler32 (adler32-zlib) [ 5882.503671] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 5882.907765] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 5884.567718] Key type lgssc registered [ 5885.656351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5893.950507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5893.986558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.799029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.621748] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5905.902578] LustreError: 184643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.924406] LustreError: 184643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5911.019622] LustreError: 184644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.116242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.558771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5916.566826] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5924.815918] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5931.251719] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5939.489343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.963458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.996745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5946.760681] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 5956.143870] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5977.056378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5977.072306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.084179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.117859] Lustre: Skipped 1 previous similar message [ 5981.868943] Lustre: server umount lustre-OST0000 complete [ 5987.298809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.333083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.343951] Lustre: Skipped 2 previous similar messages [ 5992.292929] Lustre: server umount lustre-MDT0000 complete [ 5992.420274] LustreError: 184644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.454742] LustreError: 184644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5996.393720] LustreError: 187247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786022844 with bad export cookie 14996356235082460815 [ 5996.406162] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5996.413917] LustreError: 187247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5997.173412] Lustre: server umount lustre-MDT0001 complete [ 6012.245277] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 6016.906169] Key type lgssc unregistered [ 6017.110723] LNet: 188201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6017.115166] LNetError: 188201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6017.135460] LNet: Removed LNI 192.168.201.138@tcp [ 6017.868111] Key type .llcrypt unregistered [ 6017.871136] Key type ._llcrypt unregistered [ 6047.008874] Key type ._llcrypt registered [ 6047.013149] Key type .llcrypt registered [ 6047.125444] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6048.765185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6048.809823] alg: No test for adler32 (adler32-zlib) [ 6049.859451] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 6050.102516] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 6051.759210] Key type lgssc registered [ 6053.422471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6062.388954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6062.417628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.906695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6068.598604] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6073.320367] LustreError: 189619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.339398] LustreError: 189619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6077.389717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.722833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6082.488502] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6090.653152] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6096.652885] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6104.781324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6105.169824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.348289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6112.041796] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6119.588196] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6132.761600] Lustre: lustre-OST0000: Not available for connect from 192.168.201.38@tcp (stopping) [ 6133.218981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6133.271936] Lustre: Skipped 1 previous similar message [ 6133.277599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6133.281206] Lustre: Skipped 1 previous similar message [ 6133.281583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.244475] Lustre: server umount lustre-OST0000 complete [ 6145.086347] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff97a250050000.ost_server_uuid 50 [ 6147.335877] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6162.827978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.841041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6162.844537] Lustre: Skipped 1 previous similar message [ 6162.875202] Lustre: Skipped 1 previous similar message [ 6168.035694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6168.047091] Lustre: Skipped 1 previous similar message [ 6169.292436] Lustre: server umount lustre-MDT0000 complete [ 6173.162834] LustreError: 189619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.202963] LustreError: 189619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6173.854169] LustreError: 189599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786023021 with bad export cookie 2414838319026238248 [ 6173.865552] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6173.869443] LustreError: 189599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6174.413601] Lustre: server umount lustre-MDT0001 complete [ 6184.800725] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:30:31 (1786023031) [ 6193.020537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.767745] LustreError: 193538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6193.880884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.567977] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6199.268430] LustreError: 193538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.380434] LustreError: 193539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6208.014350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.493342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6213.278150] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6221.332464] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6226.655443] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6234.230034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6234.584127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.718963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6241.589265] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6250.603936] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6257.937905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6258.067957] Lustre: lustre-OST0001: new disk, initializing [ 6258.072683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6258.079561] Lustre: 196383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6258.176619] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6265.102081] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6268.113993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6268.125446] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6268.240687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6277.429951] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6283.746774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6283.781459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.784465] Lustre: Skipped 1 previous similar message [ 6287.552649] Lustre: server umount lustre-OST0000 complete [ 6288.877450] LustreError: 196845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.910498] LustreError: 196845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6295.930400] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6296.191797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6301.184243] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6301.363764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6309.354809] LustreError: 195362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.397177] LustreError: 195362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6319.598148] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6319.604982] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.614507] Lustre: Skipped 1 previous similar message [ 6319.618824] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6319.623242] Lustre: Skipped 2 previous similar messages [ 6323.510382] Lustre: server umount lustre-OST0001 complete [ 6329.827457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.865519] Lustre: Skipped 2 previous similar messages [ 6334.344876] Lustre: server umount lustre-MDT0000 complete [ 6337.943680] LustreError: 194277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786023185 with bad export cookie 2414838319026239319 [ 6337.951868] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6337.956315] LustreError: 194277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6338.300529] Lustre: server umount lustre-MDT0001 complete [ 6364.122198] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:33:30 (1786023210) [ 6372.003206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.174634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6372.242966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6372.486500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.797532] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6390.391284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.461562] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6390.497648] Lustre: 199823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6396.899478] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6409.538370] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6417.949384] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6426.494663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.866743] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6435.318061] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6436.107632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6444.046508] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6451.724388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6451.799202] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6452.104137] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6452.114159] Lustre: Skipped 2 previous similar messages [ 6459.307798] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6467.045069] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6470.633693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.640120] Lustre: Skipped 1 previous similar message [ 6470.641522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6470.658304] Lustre: Skipped 3 previous similar messages [ 6475.880807] Lustre: server umount lustre-OST0000 complete [ 6478.322580] LustreError: 202076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.361931] LustreError: 202076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6496.236821] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.239736] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6496.258357] Lustre: Skipped 1 previous similar message [ 6496.266322] LustreError: Skipped 2 previous similar messages [ 6499.814669] Lustre: server umount lustre-OST0001 complete [ 6506.467536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6506.480367] Lustre: Skipped 6 previous similar messages [ 6518.240077] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6518.422908] Lustre: server umount lustre-MDT0000 complete [ 6524.244622] LustreError: 200581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786023372 with bad export cookie 2414838319026240726 [ 6524.248594] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6524.280180] LustreError: 200581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6525.160514] Lustre: server umount lustre-MDT0001 complete [ 6554.351621] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:36:41 (1786023401) [ 6571.772174] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 6583.429281] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 6627.940935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.457332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6628.602940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6628.829455] Lustre: lustre-MDT0000: new disk, initializing [ 6628.989349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.050119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6636.959633] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6651.067570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.276765] Lustre: 207928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6651.353555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6651.358114] Lustre: Skipped 1 previous similar message [ 6651.436242] Lustre: lustre-MDT0001: new disk, initializing [ 6651.567340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6651.594396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6656.570545] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6666.423061] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6672.274172] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6679.624501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6680.010495] Lustre: lustre-OST0000: new disk, initializing [ 6680.020975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6680.033208] Lustre: 209899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6687.995739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6688.015332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6688.225506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6689.086903] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6702.504317] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6709.329590] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6709.491096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6714.672590] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6714.903970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6718.947103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6718.947574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.960625] LustreError: Skipped 1 previous similar message [ 6718.967359] Lustre: Skipped 2 previous similar messages [ 6718.969314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6718.979156] Lustre: Skipped 7 previous similar messages [ 6731.232463] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6731.441529] Lustre: server umount lustre-OST0000 complete [ 6738.912685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6741.274905] Lustre: server umount lustre-MDT0000 complete [ 6744.544178] LustreError: 210074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6744.566638] LustreError: 210074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6745.151835] LustreError: 207920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786023593 with bad export cookie 2414838319026242287 [ 6745.163713] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6745.462987] Lustre: server umount lustre-MDT0001 complete [ 6753.641812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.685135] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6768.208751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.189386] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6782.684898] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6789.157469] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6796.698765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6804.134574] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6811.225737] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6815.939648] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6816.144537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6820.843404] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6821.028924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6823.908793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.911082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6823.924819] Lustre: Skipped 3 previous similar messages [ 6829.217089] Lustre: server umount lustre-OST0000 complete [ 6835.707538] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6840.136409] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6869.521539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6877.655779] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6879.146527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6879.159430] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6879.160557] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6879.191679] Lustre: Skipped 1 previous similar message [ 6885.105697] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6886.457389] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6891.764885] Lustre: server umount lustre-OST0000 complete [ 6896.608950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6896.615436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6896.621739] Lustre: Skipped 11 previous similar messages [ 6905.889809] Lustre: server umount lustre-MDT0000 complete [ 6909.712239] LustreError: 213961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786023757 with bad export cookie 2414838319026243162 [ 6909.721558] LustreError: 213961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6909.723548] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6919.329909] Lustre: server umount lustre-MDT0001 complete [ 6928.501128] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:42:55 (1786023775) [ 6934.644860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.088737] LustreError: 218085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6935.112598] LustreError: 218085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6935.179843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6935.194680] Lustre: Skipped 6 previous similar messages [ 6940.306853] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6948.828257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.755985] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 6963.449777] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6971.628884] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6983.299739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.929861] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7003.782583] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7012.042953] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7012.296318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7017.925942] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7018.236468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7027.019286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7027.148646] Lustre: lustre-OST0001: new disk, initializing [ 7027.150961] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7027.154394] Lustre: 221462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7035.001529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7035.013530] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7035.167502] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7035.934496] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7047.112925] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7052.310958] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7052.641811] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7058.130892] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7058.346439] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7060.961611] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7060.963114] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.997068] LustreError: Skipped 1 previous similar message [ 7061.055815] Lustre: Skipped 4 previous similar messages [ 7066.544522] Lustre: server umount lustre-OST0001 complete [ 7072.176958] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7072.354183] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7078.043382] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7078.268558] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7112.784502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.311628] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7118.332541] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7118.341051] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7123.906187] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7133.858659] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.766144] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7138.801161] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7144.016242] Lustre: server umount lustre-OST0001 complete [ 7163.877744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7163.880835] Lustre: Skipped 12 previous similar messages [ 7166.711590] Lustre: server umount lustre-OST0000 complete [ 7185.375425] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7185.535415] Lustre: server umount lustre-MDT0000 complete [ 7189.072965] LustreError: 218066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786024037 with bad export cookie 2414838319026244135 [ 7189.075610] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7189.084367] LustreError: 218066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7195.469409] Lustre: server umount lustre-MDT0001 complete [ 7220.000214] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:47:46 (1786024066) [ 7226.842780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7227.195806] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7227.238525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7232.177830] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7240.768348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.952177] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7241.004171] Lustre: 226706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7247.287889] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7255.758650] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7263.246968] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7270.234628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7270.444805] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7279.200308] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7286.145205] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7296.411659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7296.462196] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7304.379851] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7314.886772] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7316.615246] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7322.016408] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7322.301075] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7327.362933] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7327.628962] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7333.957219] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97a26000a800.ost_server_uuid 50 [ 7335.985459] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97a26000a800.ost_server_uuid in IDLE state after 0 sec [ 7337.197745] Lustre: Permanently deactivating lustre-OST0001 [ 7337.217404] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7341.811054] Lustre: Permanently reactivating lustre-OST0001 [ 7341.847393] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7341.859709] Lustre: Skipped 2 previous similar messages [ 7348.198311] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7348.209092] LustreError: Skipped 4 previous similar messages [ 7348.218982] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.227531] Lustre: Skipped 6 previous similar messages [ 7352.646293] Lustre: server umount lustre-OST0001 complete [ 7355.365562] LustreError: 228552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7355.392278] LustreError: 228552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 7362.674364] Lustre: server umount lustre-OST0000 complete [ 7372.699090] Lustre: server umount lustre-MDT0000 complete [ 7376.878847] LustreError: 226698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786024224 with bad export cookie 2414838319026245423 [ 7376.881473] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7376.886921] LustreError: 226698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7377.300327] Lustre: server umount lustre-MDT0001 complete [ 7410.780593] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:50:56 (1786024256) [ 7421.028379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.197677] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7421.216955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7421.221177] Lustre: Skipped 2 previous similar messages [ 7426.675321] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7435.837463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.909348] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7435.932211] Lustre: 233401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7440.929483] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7448.599316] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7454.416035] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7461.118666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7461.385172] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7461.420523] Lustre: Permanently deactivating lustre-OST0000 [ 7461.436834] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7461.653897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.663180] Lustre: Skipped 10 previous similar messages [ 7468.567798] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7468.952126] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7477.536394] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7484.271469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7484.342628] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7490.634134] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7492.848153] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7499.158825] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7501.158878] Lustre: Permanently reactivating lustre-OST0000 [ 7501.174213] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7501.183664] Lustre: Skipped 2 previous similar messages [ 7510.589303] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7510.592196] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 000000009d80bed3, cur 1786024358 deadline 1786024346 last 1786024316 [ 7510.594941] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7510.602942] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7510.635581] Lustre: Skipped 1 previous similar message [ 7510.720982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7522.229598] Lustre: server umount lustre-OST0001 complete [ 7533.955665] Lustre: server umount lustre-OST0000 complete [ 7547.256519] Lustre: server umount lustre-MDT0000 complete [ 7552.645917] LustreError: 233393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786024400 with bad export cookie 2414838319026247089 [ 7552.668594] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7553.831965] Lustre: server umount lustre-MDT0001 complete [ 7565.197697] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 7567.642356] Key type lgssc unregistered [ 7567.908147] LNet: 238284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7567.921144] LNetError: 238284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7567.961787] LNet: Removed LNI 192.168.201.138@tcp [ 7568.753814] Key type .llcrypt unregistered [ 7568.755313] Key type ._llcrypt unregistered [ 7591.872567] Key type ._llcrypt registered [ 7591.873793] Key type .llcrypt registered [ 7591.953672] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7592.873755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7592.915513] alg: No test for adler32 (adler32-zlib) [ 7593.925634] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 7594.151832] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 7595.831517] Key type lgssc registered [ 7597.199713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7607.221355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7607.242180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.135480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.566345] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7618.531565] LustreError: 239764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7618.545186] LustreError: 239764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7619.559702] LustreError: 239763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7623.785596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.921156] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7623.981538] Lustre: 239751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7623.997458] Lustre: Permanently deactivating lustre-MDT0001 [ 7624.068914] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7624.325231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7629.946737] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7633.427301] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7638.243604] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7644.470974] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7652.736808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7653.104553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7654.123623] LustreError: 241667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7654.159890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7659.494590] LustreError: 241606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.338643] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7664.632441] LustreError: 241667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.872968] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7674.852753] LustreError: 241607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7674.897522] LustreError: 241607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7676.104075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7676.410437] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7681.549044] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7681.554712] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7684.545396] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7694.841467] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7699.488310] Lustre: Permanently reactivating lustre-MDT0001 [ 7699.535933] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7699.549322] Lustre: Skipped 2 previous similar messages [ 7705.116317] Lustre: lustre-MDT0001: Client bd313905-6f26-471e-8548-a8eb56bf5d06 (at 192.168.201.38@tcp) reconnecting [ 7705.680230] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.687852] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7705.692747] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7705.699825] LustreError: 243332:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7705.703300] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7705.708466] LustreError: 243332:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7705.714657] LustreError: 243332:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.138@tcp: cfg command failed: rc = -108 [ 7705.718935] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7705.718935] [ 7705.722383] LustreError: 239755:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7710.529195] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7712.429783] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7717.641183] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7717.907223] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7722.985097] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7723.198167] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7725.903585] Lustre: Permanently deactivating lustre-MDT0001 [ 7725.913126] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7725.915924] Lustre: Skipped 2 previous similar messages [ 7731.865679] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7741.921607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7741.934032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.936604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7741.978525] Lustre: Skipped 3 previous similar messages [ 7746.226100] Lustre: server umount lustre-MDT0000 complete [ 7747.045158] LustreError: 239764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7747.061300] LustreError: 239764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7749.823410] LustreError: 239743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786024597 with bad export cookie 31774277806631363 [ 7749.834835] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.838849] LustreError: 239743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7750.361476] Lustre: server umount lustre-MDT0001 complete [ 7756.379416] Lustre: server umount lustre-OST0000 complete [ 7767.780800] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:56:54 (1786024614) [ 7779.741281] Lustre: server umount lustre-OST0001 complete [ 7787.546566] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 7799.446069] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 7835.090444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.253775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7835.259028] Lustre: Skipped 2 previous similar messages [ 7835.274598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7835.333550] Lustre: lustre-MDT0000: new disk, initializing [ 7835.384705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.395754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7839.926396] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7853.097127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.270175] Lustre: 248026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7853.375587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7853.379380] Lustre: Skipped 1 previous similar message [ 7853.430913] Lustre: lustre-MDT0001: new disk, initializing [ 7853.524810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7853.568946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7853.582367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7858.253552] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7867.703509] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7874.393143] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7882.020297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7882.303224] Lustre: lustre-OST0000: new disk, initializing [ 7882.312694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7882.324939] Lustre: 249996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7882.493584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7887.960878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7887.967364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7888.139741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7890.386834] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7901.027279] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7904.116592] LustreError: 248036:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7914.119256] LustreError: 248036:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7914.172610] LustreError: 248034:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7914.498451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7914.791176] Lustre: lustre-OST0001: Not available for connect from 192.168.201.38@tcp (not set up) [ 7914.795193] Lustre: Skipped 3 previous similar messages [ 7914.803859] Lustre: lustre-OST0001: new disk, initializing [ 7914.820425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7914.840328] Lustre: 251108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7915.105906] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7923.065139] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 7924.208490] LustreError: 248034:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7924.848077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7924.865587] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7924.963772] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7933.039822] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7934.786373] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7940.595564] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7940.595637] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.601722] LustreError: Skipped 1 previous similar message [ 7940.614597] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7940.627802] Lustre: Skipped 1 previous similar message [ 7942.842260] Lustre: server umount lustre-OST0001 complete [ 7945.726573] LustreError: 249981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.762965] LustreError: 249981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7961.065672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7961.074224] Lustre: Skipped 2 previous similar messages [ 7961.081488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7961.086840] Lustre: Skipped 1 previous similar message [ 7966.181356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7966.697788] Lustre: server umount lustre-OST0000 complete [ 7971.299621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7971.301186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7971.315641] Lustre: Skipped 1 previous similar message [ 7976.430867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7976.436187] Lustre: Skipped 3 previous similar messages [ 7977.308178] Lustre: server umount lustre-MDT0000 complete [ 7981.571239] LustreError: 248019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786024829 with bad export cookie 31774277806633960 [ 7981.576686] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7981.579724] LustreError: 248019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7987.978649] Lustre: server umount lustre-MDT0001 complete [ 8000.091814] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8002.650774] Key type lgssc unregistered [ 8002.865805] LNet: 253144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8002.882402] LNetError: 253144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8002.893917] LNet: Removed LNI 192.168.201.138@tcp [ 8004.012728] Key type .llcrypt unregistered [ 8004.017990] Key type ._llcrypt unregistered [ 8045.694264] Key type ._llcrypt registered [ 8045.696215] Key type .llcrypt registered [ 8045.831042] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8047.711223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8047.735426] alg: No test for adler32 (adler32-zlib) [ 8048.867807] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 8049.230737] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8050.914490] Key type lgssc registered [ 8051.943197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8058.744794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8058.776853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8060.090571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8060.109984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8060.254451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8064.753654] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8073.736071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8073.835092] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8073.874729] Lustre: 255364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8074.136406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8078.660777] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8086.179955] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8092.390764] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8100.396596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8100.715655] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8100.994865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8108.764756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8109.053376] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8117.554847] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8180.204637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8180.231070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8180.256896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8180.719559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8180.741601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8180.779913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8183.512846] Lustre: server umount lustre-OST0000 complete [ 8188.280761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8199.224517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8199.657251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8200.698160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8200.705765] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8200.722253] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8207.932436] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8218.229434] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8262.123437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8262.126637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8262.141920] Lustre: Skipped 1 previous similar message [ 8262.151404] Lustre: Skipped 1 previous similar message [ 8265.210850] Lustre: server umount lustre-OST0000 complete [ 8272.353823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.385813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8272.402290] Lustre: Skipped 1 previous similar message [ 8276.018787] Lustre: server umount lustre-MDT0000 complete [ 8281.064732] LustreError: 255357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025129 with bad export cookie 6473249201822636793 [ 8281.070860] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8281.073281] LustreError: 255357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8281.506401] Lustre: server umount lustre-MDT0001 complete [ 8297.153483] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8301.141820] Key type lgssc unregistered [ 8301.391051] LNet: 260800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8301.394798] LNetError: 260800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8301.412337] LNet: Removed LNI 192.168.201.138@tcp [ 8302.584230] Key type .llcrypt unregistered [ 8302.586683] Key type ._llcrypt unregistered [ 8332.353224] Key type ._llcrypt registered [ 8332.355264] Key type .llcrypt registered [ 8332.549701] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8333.548641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8333.600536] alg: No test for adler32 (adler32-zlib) [ 8334.784266] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 8335.182801] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8336.975428] Key type lgssc registered [ 8339.700990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8349.992397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8350.009517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8351.797744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8357.477088] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8361.955582] LustreError: 262217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8361.976952] LustreError: 262217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8367.078375] LustreError: 262218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.253278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.723850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8372.947463] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8380.127487] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8387.544750] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8398.283557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8399.226932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.108493] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8416.765686] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8421.765620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8452.576484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8452.584657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8452.587091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8452.600652] Lustre: Skipped 1 previous similar message [ 8456.872959] Lustre: server umount lustre-OST0000 complete [ 8462.817412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.824357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8462.840237] Lustre: Skipped 1 previous similar message [ 8462.862040] Lustre: Skipped 2 previous similar messages [ 8467.940551] LustreError: 262214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8467.960982] LustreError: 262214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8467.977629] Lustre: server umount lustre-MDT0000 complete [ 8471.168810] LustreError: 262200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025319 with bad export cookie 7268959097315472496 [ 8471.180068] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8471.564945] Lustre: server umount lustre-MDT0001 complete [ 8486.111175] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8488.860388] Key type lgssc unregistered [ 8489.269625] LNet: 266527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8489.274781] LNetError: 266527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8489.297285] LNet: Removed LNI 192.168.201.138@tcp [ 8490.473266] Key type .llcrypt unregistered [ 8490.475462] Key type ._llcrypt unregistered [ 8511.289580] Key type ._llcrypt registered [ 8511.292411] Key type .llcrypt registered [ 8511.474847] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8513.158470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8513.272340] alg: No test for adler32 (adler32-zlib) [ 8514.402232] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 8514.673339] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8516.416689] Key type lgssc registered [ 8517.806540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8526.224428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8526.250491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.654538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8531.465809] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8537.066934] LustreError: 267750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.095700] LustreError: 267750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8541.340581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.722358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8547.176605] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8556.925309] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8565.175763] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8574.763383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8575.450914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8581.313380] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8588.672789] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8598.037205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8612.257971] Lustre: server umount lustre-OST0000 complete [ 8618.465893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8618.473996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8618.484078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8618.506833] Lustre: Skipped 1 previous similar message [ 8618.517124] Lustre: Skipped 1 previous similar message [ 8623.587039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8623.592245] Lustre: Skipped 1 previous similar message [ 8628.703442] Lustre: 267064:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786025460/real 1786025460] req@000000000e6db122 x1872783330017792/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786025476 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8628.704993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8628.723537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.741810] Lustre: Skipped 1 previous similar message [ 8629.729067] Lustre: 267063:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786025461/real 1786025461] req@000000006b50d435 x1872783330018560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786025477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8630.244119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8630.506535] Lustre: server umount lustre-MDT0000 complete [ 8633.824932] LustreError: 267747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8633.838679] LustreError: 267747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8634.847313] Lustre: 267064:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786025466/real 1786025466] req@00000000a160778e x1872783330020096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786025482 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8635.072511] LustreError: 267732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025483 with bad export cookie 14643104891471900418 [ 8635.082154] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8635.099782] LustreError: 267732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8635.549650] Lustre: server umount lustre-MDT0001 complete [ 8648.949388] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8651.138180] Key type lgssc unregistered [ 8651.421952] LNet: 271451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8651.432659] LNetError: 271451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8651.467393] LNet: Removed LNI 192.168.201.138@tcp [ 8652.231692] Key type .llcrypt unregistered [ 8652.233696] Key type ._llcrypt unregistered [ 8683.693950] Key type ._llcrypt registered [ 8683.698254] Key type .llcrypt registered [ 8683.880433] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8685.159511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8685.462121] alg: No test for adler32 (adler32-zlib) [ 8686.700581] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 8687.031761] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8688.839404] Key type lgssc registered [ 8690.737782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8701.506940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8701.527607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.993569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8709.661935] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8712.169049] LustreError: 272872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8712.191324] LustreError: 272872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8717.297437] LustreError: 272873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8720.364848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8720.886518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8726.883578] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8742.872321] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8751.517398] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8760.086634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8760.515709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.648688] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8773.800566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8774.902135] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8801.760371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8801.763855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8801.786551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8804.836154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8804.837658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8804.854425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.898311] Lustre: server umount lustre-OST0000 complete [ 8813.028087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8813.037647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.049815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8813.054124] Lustre: Skipped 1 previous similar message [ 8815.073940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.096066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8815.098498] Lustre: Skipped 1 previous similar message [ 8817.764947] Lustre: server umount lustre-MDT0000 complete [ 8820.192484] LustreError: 273815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8820.210472] LustreError: 273815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8821.830809] LustreError: 274717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025669 with bad export cookie 3216873831124956027 [ 8821.831085] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8821.839668] LustreError: 274717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8822.370152] Lustre: server umount lustre-MDT0001 complete [ 8833.674696] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8836.590684] Key type lgssc unregistered [ 8836.906704] LNet: 277182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8836.914295] LNetError: 277182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8836.935797] LNet: Removed LNI 192.168.201.138@tcp [ 8837.897826] Key type .llcrypt unregistered [ 8837.900099] Key type ._llcrypt unregistered [ 8854.666553] Key type ._llcrypt registered [ 8854.671957] Key type .llcrypt registered [ 8854.825534] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 8855.919519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8855.942397] alg: No test for adler32 (adler32-zlib) [ 8857.106823] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 8857.275250] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 8858.903262] Key type lgssc registered [ 8860.023517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8869.116586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8869.150650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.846912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8877.027861] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8880.097906] LustreError: 278427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8880.122879] LustreError: 278427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8885.230759] LustreError: 278429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8886.388809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8886.728692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8891.595453] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8899.536294] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8905.235908] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8913.595559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8913.813893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8915.918471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8921.967549] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 8932.526981] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8946.656449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8946.670181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8946.685535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8947.683903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.685485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8947.698815] Lustre: Skipped 1 previous similar message [ 8951.621354] Lustre: server umount lustre-OST0000 complete [ 8957.923500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8957.946685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8957.949034] Lustre: Skipped 1 previous similar message [ 8962.375469] Lustre: server umount lustre-MDT0000 complete [ 8963.072176] LustreError: 278418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8963.098315] LustreError: 278418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8967.222977] LustreError: 278409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8967.301231] LustreError: 278409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8970.345676] LustreError: 278393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786025818 with bad export cookie 17052859370840412517 [ 8970.359272] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8971.443482] Lustre: server umount lustre-MDT0001 complete [ 8990.251903] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 8993.441510] Key type lgssc unregistered [ 8993.767183] LNet: 282132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8993.802420] LNetError: 282132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8993.848605] LNet: Removed LNI 192.168.201.138@tcp [ 8995.458532] Key type .llcrypt unregistered [ 8995.465739] Key type ._llcrypt unregistered [ 9010.180654] Key type ._llcrypt registered [ 9010.182335] Key type .llcrypt registered [ 9010.302535] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9028.545443] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 9049.768931] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 9051.304407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9051.316459] alg: No test for adler32 (adler32-zlib) [ 9052.310951] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 9052.530113] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 9054.175184] Key type lgssc registered [ 9055.266018] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9098.199711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9098.261595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9099.587557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9099.626314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9099.810554] Lustre: lustre-MDT0000: new disk, initializing [ 9099.953391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.985139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9106.412353] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9120.432286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9120.575026] Lustre: 285687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9120.622845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9120.626304] Lustre: Skipped 1 previous similar message [ 9120.690319] Lustre: lustre-MDT0001: new disk, initializing [ 9120.799394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9120.836405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9120.851426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9126.870720] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9136.488490] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9142.396121] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9149.235656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9149.691517] Lustre: lustre-OST0000: new disk, initializing [ 9149.696457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9149.705576] Lustre: 287661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9149.804731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9157.816799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9157.843394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9158.041430] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9158.097175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9169.099181] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9174.496866] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9174.707903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9180.939808] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9181.178558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9183.715905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9183.720522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.735165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9188.834538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9188.837792] Lustre: Skipped 1 previous similar message [ 9189.504321] Lustre: server umount lustre-OST0000 complete [ 9197.536464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9197.539380] LustreError: Skipped 1 previous similar message [ 9197.547537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9197.566764] Lustre: Skipped 1 previous similar message [ 9197.580839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9197.587451] Lustre: Skipped 1 previous similar message [ 9199.073976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.770575] Lustre: server umount lustre-MDT0000 complete [ 9205.469613] LustreError: 285682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026053 with bad export cookie 6070579723471777411 [ 9205.471387] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9205.490406] LustreError: 285682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9205.793656] Lustre: server umount lustre-MDT0001 complete [ 9217.699460] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:21:03 (1786026063) [ 9228.623620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.872933] LustreError: 290074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9229.901925] LustreError: 290074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9230.058602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9235.434731] LustreError: 290075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.469478] LustreError: 290075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9236.742920] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9239.533688] LustreError: 290074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9244.647837] LustreError: 290075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.040749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.458885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9254.536689] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9264.002343] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9273.446220] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9282.273852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9283.120544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9293.221601] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9305.182741] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9318.889629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9318.903749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9318.927932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9318.940771] Lustre: Skipped 2 previous similar messages [ 9322.722640] Lustre: server umount lustre-OST0000 complete [ 9330.658713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9330.664229] LustreError: Skipped 1 previous similar message [ 9330.665854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9330.676393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9330.690402] Lustre: Skipped 2 previous similar messages [ 9330.709758] Lustre: Skipped 3 previous similar messages [ 9333.720346] Lustre: server umount lustre-MDT0000 complete [ 9335.785914] LustreError: 292612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.808542] LustreError: 292612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9337.681892] LustreError: 290055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026185 with bad export cookie 6070579723471778293 [ 9337.704174] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9338.272546] Lustre: server umount lustre-MDT0001 complete [ 9350.811242] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 9354.552759] Key type lgssc unregistered [ 9354.953953] LNet: 293614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9354.959709] LNetError: 293614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9354.982232] LNet: Removed LNI 192.168.201.138@tcp [ 9356.187987] Key type .llcrypt unregistered [ 9356.197576] Key type ._llcrypt unregistered [ 9391.792934] Key type ._llcrypt registered [ 9391.798585] Key type .llcrypt registered [ 9392.000602] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 9411.603836] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 9413.005760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9413.027773] alg: No test for adler32 (adler32-zlib) [ 9414.469645] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 9415.082726] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 9416.855336] Key type lgssc registered [ 9418.697942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9474.968130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9474.990693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.361447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9476.381632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9476.442556] Lustre: lustre-MDT0000: new disk, initializing [ 9476.510931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.522238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9481.703069] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9492.644736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9492.783943] Lustre: 297444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9492.885616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9492.889739] Lustre: Skipped 1 previous similar message [ 9492.969079] Lustre: lustre-MDT0001: new disk, initializing [ 9493.089121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9493.133113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9493.146712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9497.928860] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9508.541949] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9514.412431] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9522.244506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9522.796413] Lustre: lustre-OST2710: new disk, initializing [ 9522.799801] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9522.805809] Lustre: 299412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9522.887932] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9530.482484] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9532.519818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9532.527362] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9532.687619] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9543.544503] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9550.964731] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9551.147577] Lustre: lustre-OST03e8: new disk, initializing [ 9551.156492] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9551.167301] Lustre: 300476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9551.252282] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9557.274598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9557.290819] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9557.412473] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9559.003580] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9568.652278] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9577.855433] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9578.363440] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9585.404691] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9585.682345] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9593.690628] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9594.102514] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9602.470937] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9602.856329] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9608.672638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9608.688291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9608.703020] Lustre: Skipped 2 previous similar messages [ 9608.715077] Lustre: Skipped 3 previous similar messages [ 9613.798484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9613.805953] Lustre: Skipped 1 previous similar message [ 9614.642523] Lustre: server umount lustre-MDT0000 complete [ 9618.598429] LustreError: 297437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026466 with bad export cookie 4625301624319144880 [ 9618.604219] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9618.623561] LustreError: 297437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9618.916328] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9618.934631] Lustre: Skipped 2 previous similar messages [ 9619.043383] Lustre: server umount lustre-MDT0001 complete [ 9626.119262] Lustre: server umount lustre-OST2710 complete [ 9631.326902] Lustre: server umount lustre-OST03e8 complete [ 9652.512240] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [ 9662.662354] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 9700.326973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9700.541544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9700.563313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9700.638398] Lustre: lustre-MDT0000: new disk, initializing [ 9700.712371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9700.734965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9704.809427] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9716.821488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.935786] Lustre: 305729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9716.977582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9716.980318] Lustre: Skipped 1 previous similar message [ 9717.042635] Lustre: lustre-MDT0001: new disk, initializing [ 9717.112156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9717.147316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9717.161322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9722.104680] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9733.303884] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9739.178527] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9745.641694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9745.839497] Lustre: lustre-OST0000: new disk, initializing [ 9745.843584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9745.848270] Lustre: 307695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9745.912722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9751.858736] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9756.248369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9756.261809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9756.364831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9760.811670] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9765.796514] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9766.060454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9770.460570] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9770.695284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9777.120230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9777.125288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9777.128236] LustreError: Skipped 1 previous similar message [ 9777.129926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9777.145490] Lustre: Skipped 1 previous similar message [ 9778.864932] Lustre: server umount lustre-OST0000 complete [ 9783.777704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9783.789529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.809636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9783.822410] Lustre: Skipped 1 previous similar message [ 9789.216481] Lustre: server umount lustre-MDT0000 complete [ 9792.480599] LustreError: 305735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.500539] LustreError: 305735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9793.346621] LustreError: 305721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026641 with bad export cookie 4625301624319146567 [ 9793.360615] LustreError: 305721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9793.367076] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9793.859760] Lustre: server umount lustre-MDT0001 complete [ 9803.652389] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:30:50 (1786026650) [ 9805.372688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9806.948959] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:30:53 (1786026653) [ 9809.835158] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 9838.774857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.110868] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9839.130758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9839.260247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9842.780774] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9852.078148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9852.162689] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9852.230885] Lustre: 311550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9857.652775] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [ 9879.334542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.502570] Lustre: Denying initial registration attempt from nid 192.168.201.138@tcp, specified as failover [ 9879.506472] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9879.509493] LustreError: 313006:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9879.519353] LustreError: 313006:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9879.522104] LustreError: 313006:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9879.556517] Lustre: server umount lustre-OST0000 complete [ 9879.558511] LustreError: 313006:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9888.227341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9888.233298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9888.239715] Lustre: Skipped 1 previous similar message [ 9888.248066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9888.250244] Lustre: Skipped 2 previous similar messages [ 9893.435590] Lustre: server umount lustre-MDT0000 complete [ 9896.930617] LustreError: 312463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.947387] LustreError: 312463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9898.420515] LustreError: 311544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026746 with bad export cookie 4625301624319147442 [ 9898.423537] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9898.448081] LustreError: 311544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9899.360250] Lustre: server umount lustre-MDT0001 complete [ 9913.269962] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [ 9916.472322] Key type lgssc unregistered [ 9916.936963] LNet: 314126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9916.946407] LNetError: 314126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9917.992461] LNet: Removed LNI 192.168.201.138@tcp [ 9918.940558] Key type .llcrypt unregistered [ 9918.943275] Key type ._llcrypt unregistered [ 9933.129905] Key type ._llcrypt registered [ 9933.133332] Key type .llcrypt registered [ 9933.304072] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 9934.657848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9934.794161] alg: No test for adler32 (adler32-zlib) [ 9935.867233] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 9936.197364] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [ 9937.935496] Key type lgssc registered [ 9939.727972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9989.084134] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [ 9999.861583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9999.882960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10001.113859] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10001.138488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10001.282970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10005.786716] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10016.254471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10016.371914] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10016.414274] Lustre: 316893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10016.769690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10020.604869] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10027.919600] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10033.647314] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10053.226078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10053.423204] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10053.659418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10060.209420] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10068.498215] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10071.018833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10071.025730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.039225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10071.525563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10076.641191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10076.649891] Lustre: Skipped 1 previous similar message [10076.748548] Lustre: server umount lustre-OST0000 complete [10081.762062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.768025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10081.778556] Lustre: Skipped 2 previous similar messages [10081.785325] Lustre: Skipped 1 previous similar message [10086.652677] Lustre: server umount lustre-MDT0000 complete [10090.463310] LustreError: 316885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786026938 with bad export cookie 9372396043316869248 [10090.472418] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10090.478632] LustreError: 316885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10090.800875] Lustre: server umount lustre-MDT0001 complete [10102.578801] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [10105.071287] Key type lgssc unregistered [10105.380109] LNet: 320468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10105.387321] LNetError: 320468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10105.405967] LNet: Removed LNI 192.168.201.138@tcp [10106.412978] Key type .llcrypt unregistered [10106.414646] Key type ._llcrypt unregistered [10132.173678] Key type ._llcrypt registered [10132.178152] Key type .llcrypt registered [10132.283727] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [10133.583550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10133.728447] alg: No test for adler32 (adler32-zlib) [10135.256744] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [10135.745758] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [10137.583887] Key type lgssc registered [10139.050810] Lustre: Echo OBD driver; http://www.lustre.org/ [10151.031842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10151.047443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10152.844796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10157.824221] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10162.145593] LustreError: 321886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10162.185588] LustreError: 321886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10167.275881] LustreError: 321887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10167.548241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10167.889284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10173.310597] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10181.339089] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10186.481923] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10196.214043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10196.969973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10201.954648] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10210.703076] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10222.564363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10222.567934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10222.570507] Lustre: Skipped 2 previous similar messages [10222.577063] Lustre: Skipped 2 previous similar messages [10226.331352] Lustre: server umount lustre-MDT0000 complete [10227.682195] LustreError: 321887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.791116] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027078 with bad export cookie 4960087361587140859 [10230.804245] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10230.811883] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10232.800674] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10232.801727] LustreError: 321883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.820404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10232.820412] Lustre: Skipped 1 previous similar message [10232.853445] LustreError: 321883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10235.812457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10235.831548] Lustre: Skipped 1 previous similar message [10237.233357] Lustre: server umount lustre-MDT0001 complete [10244.177142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10255.865846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.363945] LustreError: 323720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.381212] LustreError: 323720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10276.490588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10276.561126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10280.636671] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10286.061668] LustreError: 325480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.082903] LustreError: 325480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10290.156978] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10292.473991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.233747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10298.366930] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10299.580078] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10309.278224] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10310.758851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10316.535654] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10318.033436] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10323.937235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10323.960980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10323.961542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10323.974512] Lustre: Skipped 1 previous similar message [10329.076101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10329.080763] Lustre: Skipped 1 previous similar message [10329.837352] Lustre: server umount lustre-OST0000 complete [10334.178798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10334.191935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10334.213902] Lustre: Skipped 2 previous similar messages [10339.299211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10339.312061] Lustre: Skipped 3 previous similar messages [10340.335211] Lustre: server umount lustre-MDT0000 complete [10344.136596] LustreError: 321867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027192 with bad export cookie 4960087361587148034 [10344.146065] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10344.534778] Lustre: server umount lustre-MDT0001 complete [10357.552673] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [10361.353584] Key type lgssc unregistered [10361.775966] LNet: 328159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10361.807286] LNetError: 328159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10362.859725] LNet: Removed LNI 192.168.201.138@tcp [10363.894222] Key type .llcrypt unregistered [10363.897318] Key type ._llcrypt unregistered [10389.648913] Key type ._llcrypt registered [10389.650783] Key type .llcrypt registered [10389.770660] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [10391.397815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10391.553576] alg: No test for adler32 (adler32-zlib) [10392.596891] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [10392.794596] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [10394.479207] Key type lgssc registered [10396.333721] Lustre: Echo OBD driver; http://www.lustre.org/ [10404.420775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10404.443339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.111449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10410.739940] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10416.615685] LustreError: 329578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.642704] LustreError: 329578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10417.635647] LustreError: 329577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10421.705563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.083392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10428.774097] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10438.516905] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10446.254679] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10451.938465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10451.941579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10451.969020] Lustre: Skipped 1 previous similar message [10451.985308] Lustre: Skipped 1 previous similar message [10456.752363] Lustre: server umount lustre-MDT0000 complete [10457.064851] LustreError: 329574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10457.103026] LustreError: 329574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10460.806890] LustreError: 329558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027308 with bad export cookie 12219737231327493326 [10460.818037] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10461.040298] Lustre: server umount lustre-MDT0001 complete [10475.606645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10475.831460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10475.862202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10476.041663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10480.345808] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10488.115727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10488.188658] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10488.213660] Lustre: 332168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10488.442703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10492.739184] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10500.906836] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10507.748934] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10517.819720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10518.116278] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10518.128546] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10518.139754] LustreError: 334053:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10518.144830] LustreError: 334053:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10518.149410] LustreError: 334053:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10518.203084] Lustre: server umount lustre-OST0000 complete [10518.205402] LustreError: 334053:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10528.810129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10528.999549] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10529.139433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10537.223373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10540.916733] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10552.456200] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10561.728849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.947170] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10562.139977] Lustre: lustre-OST0001: new disk, initializing [10562.147384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10562.159972] Lustre: 335595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10562.371488] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10569.770313] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10569.849658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10569.863517] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10569.986981] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10577.731857] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10585.573960] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10585.589096] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10585.596306] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10585.607523] Lustre: Skipped 1 previous similar message [10585.623949] Lustre: Skipped 1 previous similar message [10587.437524] Lustre: server umount lustre-OST0001 complete [10590.717484] LustreError: 334518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10590.748791] LustreError: 334518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10593.764697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10593.764789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10593.774621] LustreError: Skipped 2 previous similar messages [10593.779233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10593.792557] Lustre: Skipped 1 previous similar message [10593.819293] Lustre: Skipped 1 previous similar message [10595.815540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10595.824147] LustreError: 334521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10595.826823] Lustre: Skipped 1 previous similar message [10595.865385] LustreError: 334521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10597.185223] Lustre: server umount lustre-OST0000 complete [10606.048835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10606.053512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10606.065365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10606.080246] Lustre: Skipped 1 previous similar message [10606.092600] Lustre: Skipped 1 previous similar message [10607.640812] Lustre: server umount lustre-MDT0000 complete [10611.168731] LustreError: 332175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.177865] LustreError: 332175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10611.281110] LustreError: 332162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027459 with bad export cookie 12219737231327493879 [10611.288406] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10611.300886] LustreError: 332162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10611.850586] Lustre: server umount lustre-MDT0001 complete [10624.926987] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [10628.692518] Key type lgssc unregistered [10629.111783] LNet: 337555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10629.120860] LNetError: 337555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10629.139501] LNet: Removed LNI 192.168.201.138@tcp [10630.287420] Key type .llcrypt unregistered [10630.290777] Key type ._llcrypt unregistered [10720.062590] Key type ._llcrypt registered [10720.065067] Key type .llcrypt registered [10720.233918] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [10738.438821] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [10739.863285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10739.995069] alg: No test for adler32 (adler32-zlib) [10741.051376] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [10741.321127] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [10742.999172] Key type lgssc registered [10744.628368] Lustre: Echo OBD driver; http://www.lustre.org/ [10789.045934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10789.087523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10790.614249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10790.686831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10790.813793] Lustre: lustre-MDT0000: new disk, initializing [10790.998169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10791.063402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10798.417781] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10811.971662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10812.175234] Lustre: 342496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10812.278605] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10812.283107] Lustre: Skipped 1 previous similar message [10812.440770] Lustre: lustre-MDT0001: new disk, initializing [10812.634595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10812.720409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10812.734917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10818.959895] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10830.726216] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10836.808531] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10844.351405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10844.646865] Lustre: lustre-OST0000: new disk, initializing [10844.649449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10844.662841] Lustre: 344465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10844.796666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10851.464222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10851.485749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10851.657296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10854.179965] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10867.660659] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10872.986056] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10873.364307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10879.257748] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10879.595491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10882.536629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10882.538173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10882.557523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10882.568295] LustreError: Skipped 1 previous similar message [10887.678457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10887.698347] Lustre: Skipped 2 previous similar messages [10888.845585] Lustre: server umount lustre-OST0000 complete [10894.822616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10894.832965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.858844] Lustre: Skipped 1 previous similar message [10894.881279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10897.891367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10897.893761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10901.162675] Lustre: server umount lustre-MDT0000 complete [10906.335123] LustreError: 342490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786027754 with bad export cookie 1511446966378669129 [10906.361234] LustreError: 342490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10906.361929] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10906.977669] Lustre: server umount lustre-MDT0001 complete [10918.978489] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:49:25 (1786027765) [10929.476752] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:49:36 (1786027776) [10940.003492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10940.481183] LustreError: 347222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.507514] LustreError: 347222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10940.560592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10945.781917] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10946.016686] LustreError: 347223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.116671] LustreError: 347222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.235242] LustreError: 347223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.673131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10956.207323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10961.587680] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [10970.965289] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10978.830430] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10986.879320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10987.096487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10995.851717] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11007.658248] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11022.134436] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11024.753125] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11026.873659] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11029.414191] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11032.067103] Lustre: Failing over lustre-MDT0000 [11032.440344] Lustre: server umount lustre-MDT0000 complete [11033.056500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11033.082108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11033.100071] LustreError: 347952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.768903] LustreError: 347218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.811113] LustreError: 347218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11053.024131] Lustre: 340718:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786027884/real 1786027884] req@0000000007029aa1 x1872785665244032/t0(0) o400->MGC192.168.201.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786027900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11053.039440] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11054.234427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11058.144789] LustreError: 348808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.167683] LustreError: 348808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [11063.277732] LustreError: 340715:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000256e7b1e x1872785665250048/t0(0) o250->MGC192.168.201.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11063.626661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11063.682937] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11064.815757] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11068.912026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11068.970967] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11069.091313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11069.791329] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11082.742925] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11085.391546] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11089.561934] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11091.889296] Lustre: Failing over lustre-MDT0000 [11092.534776] Lustre: server umount lustre-MDT0000 complete [11094.511200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.516108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11094.543837] Lustre: Skipped 3 previous similar messages [11094.546771] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11094.546783] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11110.879279] Lustre: 340719:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786027942/real 1786027942] req@00000000551b1474 x1872785665266432/t0(0) o400->MGC192.168.201.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786027958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11110.926718] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11112.733447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.525529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11120.601320] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11121.151901] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11125.761392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11125.779620] Lustre: Skipped 2 previous similar messages [11125.818211] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11125.874222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11126.182984] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11138.892459] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11140.994712] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11151.330777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11151.337252] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11151.344431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11151.359698] Lustre: Skipped 1 previous similar message [11151.369410] Lustre: Skipped 1 previous similar message [11155.203722] Lustre: server umount lustre-MDT0000 complete [11159.924882] LustreError: 347204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028007 with bad export cookie 1511446966378671838 [11159.927263] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11159.935843] LustreError: 347204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11160.451236] Lustre: server umount lustre-MDT0001 complete [11175.501283] Lustre: server umount lustre-OST0000 complete [11198.080668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11198.821446] LustreError: 353601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.871048] LustreError: 353601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [11198.996916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11204.436341] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11216.367654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11223.637732] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11234.080813] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11240.750442] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11248.803173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11257.553420] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11266.584525] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11269.755571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11280.352549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11280.359514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11280.371460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11280.385394] Lustre: Skipped 2 previous similar messages [11280.409843] Lustre: Skipped 3 previous similar messages [11286.191527] Lustre: server umount lustre-OST0000 complete [11297.605534] Lustre: server umount lustre-MDT0000 complete [11301.883904] LustreError: 353581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028149 with bad export cookie 1511446966378672587 [11301.895443] LustreError: 353581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11301.901970] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11302.548671] Lustre: server umount lustre-MDT0001 complete [11315.353480] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [11319.776599] Key type lgssc unregistered [11320.224573] LNet: 357151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11320.240820] LNetError: 357151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11320.267917] LNet: Removed LNI 192.168.201.138@tcp [11321.442149] Key type .llcrypt unregistered [11321.443789] Key type ._llcrypt unregistered [11356.801354] Key type ._llcrypt registered [11356.804688] Key type .llcrypt registered [11356.909581] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [11358.697654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11358.761185] alg: No test for adler32 (adler32-zlib) [11360.046245] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [11360.523359] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [11362.448141] Key type lgssc registered [11363.997144] Lustre: Echo OBD driver; http://www.lustre.org/ [11374.103753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11374.127302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11375.585399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11381.128864] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11385.828583] LustreError: 358676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11385.854943] LustreError: 358676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11390.953779] LustreError: 358675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11391.001486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11391.325533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11396.566190] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11406.779463] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11414.835051] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11424.481046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11425.120378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11433.146119] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11441.661451] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11445.654158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11461.089201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11461.094840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11461.121665] Lustre: Skipped 2 previous similar messages [11461.140752] Lustre: Skipped 2 previous similar messages [11466.222680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11471.336554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11471.341836] Lustre: Skipped 3 previous similar messages [11473.375240] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11473.792902] Lustre: server umount lustre-MDT0000 complete [11476.450637] LustreError: 358670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11476.482166] LustreError: 358670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11478.616015] LustreError: 358655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028326 with bad export cookie 15762456479983965134 [11478.623912] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11479.311363] Lustre: server umount lustre-MDT0001 complete [11493.783242] Lustre: server umount lustre-OST0000 complete [11509.304454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.847358] LustreError: 362412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.921288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11515.366248] LustreError: 362413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.161983] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11524.582633] LustreError: 362413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11524.598244] LustreError: 362413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11526.010918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11526.401269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11532.457174] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11541.814374] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11550.727891] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11559.691705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11560.133421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11567.435955] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11576.148294] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11579.567521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11598.307619] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11598.319183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11598.343924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11598.363019] Lustre: Skipped 2 previous similar messages [11599.981390] Lustre: server umount lustre-MDT0000 complete [11603.431234] LustreError: 362412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.459550] LustreError: 362412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11603.691613] LustreError: 364257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028451 with bad export cookie 15762456479983966247 [11603.694497] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11603.702510] LustreError: 364257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11604.118240] Lustre: server umount lustre-MDT0001 complete [11619.444397] Lustre: server umount lustre-OST0000 complete [11638.943903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11639.544991] LustreError: 366339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.655175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11645.254912] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11655.270967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11661.139698] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11670.151097] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11676.874920] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11686.311654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11686.803478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11686.819157] Lustre: Skipped 1 previous similar message [11694.258205] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11703.688581] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11709.356203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11719.648606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11719.653802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11719.664285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11719.669995] Lustre: Skipped 3 previous similar messages [11719.685077] Lustre: Skipped 1 previous similar message [11721.890867] Lustre: server umount lustre-OST0000 complete [11727.329977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11727.345429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11729.891347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11729.910414] Lustre: Skipped 2 previous similar messages [11733.261372] Lustre: server umount lustre-MDT0000 complete [11735.015802] LustreError: 366340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.047071] LustreError: 366340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11739.579453] LustreError: 367073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028587 with bad export cookie 15762456479983967269 [11739.585645] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11739.594577] LustreError: 367073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11740.259062] Lustre: server umount lustre-MDT0001 complete [11757.604821] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [11762.671921] Key type lgssc unregistered [11763.194061] LNet: 369888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11763.202202] LNetError: 369888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11763.235770] LNet: Removed LNI 192.168.201.138@tcp [11764.442408] Key type .llcrypt unregistered [11764.445146] Key type ._llcrypt unregistered [11796.218107] Key type ._llcrypt registered [11796.220213] Key type .llcrypt registered [11796.425525] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [11798.243499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11798.295443] alg: No test for adler32 (adler32-zlib) [11799.458985] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [11799.835215] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [11801.575965] Key type lgssc registered [11803.308853] Lustre: Echo OBD driver; http://www.lustre.org/ [11813.390521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11813.408967] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11813.420918] LustreError: 371330:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11813.445524] LustreError: 371330:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11813.456519] LustreError: 371330:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11813.471117] LustreError: 371330:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11813.486604] LustreError: 371330:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11822.917123] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11822.922310] LustreError: 371727:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11822.935480] LustreError: 371727:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11822.942671] LustreError: 371727:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11822.947478] LustreError: 371727:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11822.955570] LustreError: 371727:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11848.458709] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [11851.948711] Key type lgssc unregistered [11852.280474] LNet: 372733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11852.299314] LNetError: 372733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11852.317949] LNet: Removed LNI 192.168.201.138@tcp [11853.495627] Key type .llcrypt unregistered [11853.499972] Key type ._llcrypt unregistered [11878.468233] Key type ._llcrypt registered [11878.469554] Key type .llcrypt registered [11878.658352] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [11898.191399] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [11899.650812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11899.724064] alg: No test for adler32 (adler32-zlib) [11901.288671] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [11901.856896] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [11903.666994] Key type lgssc registered [11905.704440] Lustre: Echo OBD driver; http://www.lustre.org/ [11949.410967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11949.427824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11950.729758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11950.765715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11950.856271] Lustre: lustre-MDT0000: new disk, initializing [11951.005591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11951.050324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11957.535891] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11970.284065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.447318] Lustre: 375894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11970.509616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11970.513603] Lustre: Skipped 1 previous similar message [11970.634663] Lustre: lustre-MDT0001: new disk, initializing [11970.828191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11970.928031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11970.961624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11976.370382] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [11986.828924] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11993.633904] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12001.811818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12002.258362] Lustre: lustre-OST0000: new disk, initializing [12002.263982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12002.283179] Lustre: 377860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12002.421606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12009.629836] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12011.657869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12011.674207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12011.809658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12020.659973] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12025.092503] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12025.222325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12029.528967] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12029.743790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12032.492060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12032.501057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12032.521451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12032.527653] Lustre: Skipped 1 previous similar message [12037.602166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12037.611873] Lustre: Skipped 1 previous similar message [12042.739602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12042.745654] Lustre: Skipped 1 previous similar message [12046.303593] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12046.648389] Lustre: server umount lustre-OST0000 complete [12052.976856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12052.980115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12052.984845] Lustre: Skipped 1 previous similar message [12052.985883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12053.052902] LustreError: Skipped 1 previous similar message [12058.084530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12058.097897] Lustre: Skipped 2 previous similar messages [12066.283343] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12066.836953] Lustre: server umount lustre-MDT0000 complete [12072.245768] LustreError: 375885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786028920 with bad export cookie 11386088630223189812 [12072.248484] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12072.254825] LustreError: 375885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12072.580973] Lustre: server umount lustre-MDT0001 complete [12084.020111] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12086.260222] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:08:52 (1786028932) [12093.453809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12093.917112] LustreError: 380326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12093.951591] LustreError: 380326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12094.013920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12099.050509] LustreError: 380327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12100.017259] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12103.162379] LustreError: 380326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12108.265426] LustreError: 380327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.050680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12111.584585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12117.590493] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12127.086351] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12133.807907] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12142.227746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12142.806736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12150.859885] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12160.351707] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12167.705324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12167.837282] Lustre: lustre-OST0001: new disk, initializing [12167.840721] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12167.846898] Lustre: 383166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12167.930963] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12174.961959] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12175.508344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12175.525416] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12175.673172] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12186.529819] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12191.202376] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12191.203387] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12191.223307] LustreError: Skipped 1 previous similar message [12191.224459] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12191.224466] Lustre: Skipped 2 previous similar messages [12191.257637] Lustre: Skipped 2 previous similar messages [12196.955388] Lustre: server umount lustre-OST0001 complete [12198.891872] LustreError: 383550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.691219] LustreError: 383550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.707434] LustreError: 383550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12214.755782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12214.764063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12214.782275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12214.791834] Lustre: Skipped 4 previous similar messages [12219.577569] Lustre: server umount lustre-OST0000 complete [12227.042532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12227.054188] Lustre: Skipped 2 previous similar messages [12230.495367] Lustre: server umount lustre-MDT0000 complete [12232.161246] LustreError: 380326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12232.193013] LustreError: 380326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12235.563145] LustreError: 382172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029083 with bad export cookie 11386088630223190694 [12235.570582] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12235.577876] LustreError: 382172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12242.427309] Lustre: server umount lustre-MDT0001 complete [12256.550057] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [12260.828200] Key type lgssc unregistered [12261.332266] LNet: 385154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12261.344283] LNetError: 385154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12261.355192] LNet: Removed LNI 192.168.201.138@tcp [12262.511392] Key type .llcrypt unregistered [12262.514875] Key type ._llcrypt unregistered [12320.366651] Key type ._llcrypt registered [12320.368704] Key type .llcrypt registered [12320.534253] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [12322.109840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12322.174797] alg: No test for adler32 (adler32-zlib) [12324.437948] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [12324.986538] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [12326.655167] Key type lgssc registered [12327.856526] Lustre: Echo OBD driver; http://www.lustre.org/ [12338.354716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12338.382623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12339.668952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12339.709174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12340.025970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12345.245266] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12355.492555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12355.607555] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12355.643483] Lustre: 387956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12356.068184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12361.129561] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12369.294134] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12376.320243] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12381.155073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12381.162399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12381.174680] Lustre: Skipped 1 previous similar message [12385.341121] Lustre: server umount lustre-MDT0000 complete [12389.601593] LustreError: 387946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029237 with bad export cookie 8619204959014354816 [12389.611925] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12389.615929] LustreError: 387946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12390.268215] Lustre: server umount lustre-MDT0001 complete [12402.516264] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:14:09 (1786029249) [12409.904352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.637188] LustreError: 390469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12410.661703] LustreError: 390469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12410.725624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12415.668796] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12415.972558] LustreError: 390470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12421.094364] LustreError: 390469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.157965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12425.554398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12430.366635] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12439.483839] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12448.719189] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12457.865184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12458.091583] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12458.392862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12466.864816] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12475.366598] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12483.493384] Lustre: Permanently deactivating lustre-OST0000 [12483.533252] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12490.854473] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12493.903938] LustreError: 393460:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12493.916396] LustreError: 393460:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12493.935329] LustreError: 393460:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12497.683832] LustreError: 393508:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12497.689764] LustreError: 393508:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12506.122432] Lustre: server umount lustre-OST0000 complete [12509.213133] LustreError: 393758:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12509.219145] LustreError: 393758:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12512.736726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12512.751888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12512.771715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12512.778668] Lustre: Skipped 1 previous similar message [12518.369308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12518.377882] Lustre: Skipped 3 previous similar messages [12521.580107] Lustre: server umount lustre-MDT0000 complete [12523.503685] LustreError: 392973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12523.525872] LustreError: 392973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12525.852220] LustreError: 390450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029373 with bad export cookie 8619204959014355397 [12525.875702] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12535.491866] Lustre: server umount lustre-MDT0001 complete [12544.417344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12550.083171] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12553.233937] LustreError: 394918:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12561.155724] Lustre: 395257:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.138@tcp [12563.523730] Lustre: server umount MGS complete [12578.512471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.307642] LustreError: 395982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12579.361265] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12579.372379] Lustre: Skipped 1 previous similar message [12579.492143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12585.964313] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12590.060723] LustreError: 395983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.075665] LustreError: 395983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12595.766888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12596.123753] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12596.230234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12601.510692] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12609.999520] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12615.897846] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12618.057747] Lustre: Permanently reactivating lustre-OST0000 [12618.090859] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12618.104411] Lustre: Skipped 2 previous similar messages [12647.398986] LustreError: 397786:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12656.269759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12656.785635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12664.210906] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12673.711583] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12682.721134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12682.736071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12682.757760] Lustre: Skipped 1 previous similar message [12682.781746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12683.762838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12685.762775] Lustre: server umount lustre-OST0000 complete [12693.472521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12693.478348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12693.490137] Lustre: Skipped 1 previous similar message [12693.498436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12693.501311] Lustre: Skipped 2 previous similar messages [12696.730523] Lustre: server umount lustre-MDT0000 complete [12699.107349] LustreError: 398872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12699.129560] LustreError: 398872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12701.243594] LustreError: 395963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029549 with bad export cookie 8619204959014356447 [12701.246735] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12701.265721] LustreError: 395963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12701.542701] Lustre: server umount lustre-MDT0001 complete [12714.810974] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [12718.198277] Key type lgssc unregistered [12718.568649] LNet: 400348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12718.587492] LNetError: 400348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12718.623448] LNet: Removed LNI 192.168.201.138@tcp [12719.992563] Key type .llcrypt unregistered [12719.994066] Key type ._llcrypt unregistered [12745.532935] Key type ._llcrypt registered [12745.534658] Key type .llcrypt registered [12745.686231] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [12761.842816] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [12763.454847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12763.474483] alg: No test for adler32 (adler32-zlib) [12764.657895] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [12765.061613] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [12766.711954] Key type lgssc registered [12767.864285] Lustre: Echo OBD driver; http://www.lustre.org/ [12810.536995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12810.566248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12811.944674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12812.001453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12812.171614] Lustre: lustre-MDT0000: new disk, initializing [12812.330735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12812.367371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12817.192418] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12830.117182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12830.296587] Lustre: 403508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12830.384356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12830.395235] Lustre: Skipped 1 previous similar message [12830.523133] Lustre: lustre-MDT0001: new disk, initializing [12830.672884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12830.743360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12830.762913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12836.415948] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12849.630292] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12857.573945] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12867.976860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12868.502376] Lustre: lustre-OST0000: new disk, initializing [12868.509285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12868.518249] Lustre: 405479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12868.627738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12873.967482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12873.985967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12874.178714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12878.786826] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12889.819784] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12895.867084] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12896.077130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12901.735814] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12902.124950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12904.933703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12904.934836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12904.937479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12904.955258] Lustre: Skipped 1 previous similar message [12904.975919] LustreError: Skipped 1 previous similar message [12910.049105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12910.065886] Lustre: Skipped 2 previous similar messages [12915.171028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12915.177138] Lustre: Skipped 1 previous similar message [12919.263253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12919.628830] Lustre: server umount lustre-OST0000 complete [12925.411399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12925.435778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12925.441655] Lustre: Skipped 1 previous similar message [12930.539023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12930.727950] Lustre: server umount lustre-MDT0000 complete [12936.062083] LustreError: 403500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029784 with bad export cookie 13644130692033470663 [12936.076351] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12936.727979] Lustre: server umount lustre-MDT0001 complete [12946.972074] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12949.119267] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:23:15 (1786029795) [12957.439422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12957.979601] LustreError: 407943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12957.998603] LustreError: 407943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12958.075681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12963.311257] LustreError: 407944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12963.332108] LustreError: 407944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12964.088964] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12967.396543] LustreError: 407943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12972.523185] LustreError: 407944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.123412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12975.627871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12980.831260] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [12990.566322] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12999.750449] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13008.039251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13008.693412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13016.856890] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13027.682992] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13030.512799] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13033.553103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13041.634310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13041.665603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13041.699172] Lustre: Skipped 1 previous similar message [13041.707739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13041.717153] Lustre: Skipped 1 previous similar message [13042.663433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13045.982261] Lustre: server umount lustre-OST0000 complete [13052.386192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13052.388768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13052.407073] Lustre: Skipped 1 previous similar message [13058.020976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13058.030265] Lustre: Skipped 5 previous similar messages [13066.207228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13066.420716] Lustre: server umount lustre-MDT0000 complete [13067.236121] LustreError: 408673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13067.269029] LustreError: 408673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13071.341211] LustreError: 407924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786029919 with bad export cookie 13644130692033471538 [13071.355881] LustreError: 407924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13071.362845] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13072.301771] Lustre: server umount lustre-MDT0001 complete [13088.239855] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [13092.345978] Key type lgssc unregistered [13092.860761] LNet: 411673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13092.873994] LNetError: 411673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13093.931628] LNet: Removed LNI 192.168.201.138@tcp [13095.206525] Key type .llcrypt unregistered [13095.217776] Key type ._llcrypt unregistered [13125.609538] Key type ._llcrypt registered [13125.612203] Key type .llcrypt registered [13125.843614] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [13126.272688] Key type .llcrypt unregistered [13126.274353] Key type ._llcrypt unregistered [13151.110278] Key type ._llcrypt registered [13151.116237] Key type .llcrypt registered [13151.265492] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [13154.095756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13154.124401] alg: No test for adler32 (adler32-zlib) [13155.403983] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [13155.833717] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [13157.623640] Key type lgssc registered [13159.421943] Lustre: Echo OBD driver; http://www.lustre.org/ [13169.017262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13169.048390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13170.893632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13176.142804] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13180.388963] LustreError: 413892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13180.412244] LustreError: 413892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13185.507157] LustreError: 413893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13187.228786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13187.764107] LustreError: 413892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13187.789121] LustreError: 413892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13187.803953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13193.189693] LustreError: 413893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13193.211983] LustreError: 413893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13195.248261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13195.769398] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13200.358994] LustreError: 413893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13200.381812] LustreError: 413893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13204.760560] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13210.606602] LustreError: 413892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.630749] LustreError: 413892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13213.455236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13213.935728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13219.742870] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13229.541357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13229.551931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13229.555547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13229.556712] Lustre: Skipped 1 previous similar message [13229.597310] Lustre: Skipped 1 previous similar message [13231.471234] Lustre: server umount lustre-OST0000 complete [13239.791286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13239.797534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13239.798024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13239.820889] Lustre: Skipped 1 previous similar message [13242.898527] Lustre: server umount lustre-MDT0000 complete [13244.904191] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13244.956117] LustreError: 413888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13247.446510] LustreError: 413874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786030095 with bad export cookie 11819612488425213114 [13247.452299] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13247.457077] LustreError: 413874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13247.898875] Lustre: server umount lustre-MDT0001 complete [13260.827842] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [13264.542905] Key type lgssc unregistered [13264.954901] LNet: 417041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13264.967505] LNetError: 417041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13264.992651] LNet: Removed LNI 192.168.201.138@tcp [13266.087164] Key type .llcrypt unregistered [13266.093590] Key type ._llcrypt unregistered [13297.638092] Key type ._llcrypt registered [13297.645215] Key type .llcrypt registered [13297.751119] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [13298.856989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13299.121160] alg: No test for adler32 (adler32-zlib) [13300.397451] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [13300.796353] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [13302.543164] Key type lgssc registered [13304.262762] Lustre: Echo OBD driver; http://www.lustre.org/ [13313.656208] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13313.697815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13329.311461] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13334.432672] LustreError: 418433:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.138@tcp: failed processing log, type 4: rc = -110 [13361.183872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13369.725550] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13379.659790] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13388.870507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13389.551433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13389.641650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13394.917830] LustreError: 418459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13394.945303] LustreError: 418459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13395.108748] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13395.963970] LustreError: 418458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13395.989192] LustreError: 418458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13401.064403] LustreError: 419471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13401.090318] LustreError: 419471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13406.195580] LustreError: 418459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13406.214388] LustreError: 418459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13406.367902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13406.977797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13413.283450] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13423.301632] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13431.331415] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13441.007300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13441.015474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13441.027693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13443.045771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13443.049566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13443.072548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13445.536777] Lustre: server umount lustre-OST0000 complete [13452.711903] Lustre: server umount lustre-MDT0000 complete [13453.280351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13453.288415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13453.291504] LustreError: 419472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13453.299329] Lustre: Skipped 1 previous similar message [13453.324257] LustreError: 419472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13457.343817] LustreError: 418441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786030305 with bad export cookie 11329362314855231265 [13457.362512] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13458.119728] Lustre: server umount lustre-MDT0001 complete [13471.183667] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [13474.222411] Key type lgssc unregistered [13474.554119] LNet: 422008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13474.560641] LNetError: 422008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13474.600095] LNet: Removed LNI 192.168.201.138@tcp [13475.408599] Key type .llcrypt unregistered [13475.410744] Key type ._llcrypt unregistered [13504.431159] Key type ._llcrypt registered [13504.433918] Key type .llcrypt registered [13504.509410] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [13505.698725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13505.912069] alg: No test for adler32 (adler32-zlib) [13507.033426] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [13507.254819] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [13509.031488] Key type lgssc registered [13510.628418] Lustre: Echo OBD driver; http://www.lustre.org/ [13521.134684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13521.152680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13522.601573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13527.703818] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13532.141204] LustreError: 423443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13532.164166] LustreError: 423443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13537.263821] LustreError: 423444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13538.014939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13538.288478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13544.240254] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13554.244468] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13561.229321] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13568.524399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13569.047750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13578.576426] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13590.804580] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13592.632682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13599.727570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13599.735945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13599.767682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13602.312524] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.38@tcp (stopping) [13602.789695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13602.799708] Lustre: Skipped 1 previous similar message [13607.401241] LustreError: 424381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13608.206539] Lustre: server umount lustre-MDT0000 complete [13611.119622] LustreError: 423425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.38@tcp arrived at 1786030459 with bad export cookie 15608653540489338106 [13611.137768] LustreError: 423425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13613.027785] LustreError: 423444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13613.054732] LustreError: 423444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13613.927578] LustreError: 423424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786030461 with bad export cookie 15608653540489337441 [13613.943654] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13614.053940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13614.063141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13614.076536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13614.081627] Lustre: Skipped 3 previous similar messages [13618.145027] LustreError: 423443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13618.149502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13618.167920] LustreError: 423443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13620.203916] Lustre: server umount lustre-OST0000 complete [13628.408216] Lustre: server umount lustre-MDT0001 complete [13641.772415] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [13645.308203] Key type lgssc unregistered [13645.642939] LNet: 427132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13645.668469] LNetError: 427132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13646.696702] LNet: Removed LNI 192.168.201.138@tcp [13647.814208] Key type .llcrypt unregistered [13647.815803] Key type ._llcrypt unregistered [13677.091474] Key type ._llcrypt registered [13677.092894] Key type .llcrypt registered [13677.320386] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [13678.769594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13678.782778] alg: No test for adler32 (adler32-zlib) [13679.967708] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [13680.310088] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [13682.047875] Key type lgssc registered [13683.072439] Lustre: Echo OBD driver; http://www.lustre.org/ [13694.627577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13694.652085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13696.299111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13701.789157] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13706.724496] LustreError: 428554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13706.751382] LustreError: 428554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13711.842406] LustreError: 428555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13712.134587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13712.465992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13717.491481] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13726.104257] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13732.038835] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13739.700192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13740.126696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13741.171726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13746.959827] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13754.615870] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13762.540124] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.38@tcp (stopping) [13763.053625] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13763.082515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13763.097227] Lustre: Skipped 1 previous similar message [13766.710611] Lustre: server umount lustre-MDT0001 complete [13767.674920] LustreError: 428550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13773.297885] LustreError: 430712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13773.322707] LustreError: 430712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13777.376319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13777.399825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13777.410509] Lustre: Skipped 1 previous similar message [13777.423805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13778.401550] LustreError: 428555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13779.849372] Lustre: server umount lustre-OST0000 complete [13784.114948] Lustre: server umount lustre-MDT0000 complete [13798.495864] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [13801.321468] Key type lgssc unregistered [13801.782315] LNet: 432241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13801.790357] LNetError: 432241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13801.812280] LNet: Removed LNI 192.168.201.138@tcp [13803.087292] Key type .llcrypt unregistered [13803.090867] Key type ._llcrypt unregistered [13834.944272] Key type ._llcrypt registered [13834.952689] Key type .llcrypt registered [13835.061870] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [13850.776094] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [13852.098442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13852.190479] alg: No test for adler32 (adler32-zlib) [13853.213759] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [13853.448454] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [13855.159205] Key type lgssc registered [13856.417972] Lustre: Echo OBD driver; http://www.lustre.org/ [13898.521600] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [13916.869820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13916.910890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13918.248353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13918.290322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13918.406818] Lustre: lustre-MDT0000: new disk, initializing [13918.493881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13918.528869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13924.005372] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13942.378985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13942.819352] Lustre: lustre-OST0000: new disk, initializing [13942.832805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13942.849620] Lustre: Skipped 1 previous similar message [13942.859488] Lustre: 437279:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13942.965393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13950.998396] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13951.150906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13951.165785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13951.202736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13961.747219] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13979.687975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13979.788843] Lustre: 436261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13979.876497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13979.965335] Lustre: lustre-MDT0001: new disk, initializing [13980.137777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13980.198754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13980.203718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13984.964334] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [13995.152419] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13995.416855] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14000.497919] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14000.776353] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14010.875956] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.38@tcp (stopping) [14011.367411] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14011.375151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14011.377378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14011.400721] Lustre: Skipped 1 previous similar message [14012.531878] Lustre: server umount lustre-MDT0001 complete [14021.604728] LustreError: 436268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14021.643178] LustreError: 436268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14026.729444] LustreError: 436272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14026.749077] LustreError: 436272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14031.840852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14031.847086] LustreError: 436273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14031.869163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14031.899324] Lustre: Skipped 1 previous similar message [14036.974660] LustreError: 436272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14036.974940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14037.792102] Lustre: server umount lustre-OST0000 complete [14048.414289] Lustre: server umount lustre-MDT0000 complete [14064.077984] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [14067.802407] Key type lgssc unregistered [14068.211260] LNet: 441032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14068.223353] LNetError: 441032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14068.251600] LNet: Removed LNI 192.168.201.138@tcp [14069.349128] Key type .llcrypt unregistered [14069.351453] Key type ._llcrypt unregistered [14119.684082] Key type ._llcrypt registered [14119.692816] Key type .llcrypt registered [14119.807727] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [14120.049778] Key type .llcrypt unregistered [14120.052960] Key type ._llcrypt unregistered [14144.511904] Key type ._llcrypt registered [14144.514586] Key type .llcrypt registered [14144.710481] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [14145.755577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14145.782080] alg: No test for adler32 (adler32-zlib) [14146.909742] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [14147.437875] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [14149.159221] Key type lgssc registered [14150.593215] Lustre: Echo OBD driver; http://www.lustre.org/ [14194.185035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14194.243983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14195.692884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14195.713438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14195.772985] Lustre: lustre-MDT0000: new disk, initializing [14195.914870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14195.949474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14201.872168] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [14215.199594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14215.385786] Lustre: 444417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14215.438948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14215.443444] Lustre: Skipped 1 previous similar message [14215.552787] Lustre: lustre-MDT0001: new disk, initializing [14215.679206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14215.728666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14215.756727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14222.149200] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [14234.063235] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14240.184849] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14248.769838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14249.150852] Lustre: lustre-OST0000: new disk, initializing [14249.164864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14249.188825] Lustre: 446386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14249.366455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14257.310054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14257.339482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14257.505443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14257.532802] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [14269.935455] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14278.115268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14278.116750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14278.146362] Lustre: Skipped 2 previous similar messages [14278.171170] Lustre: Skipped 1 previous similar message [14283.233179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14283.242796] Lustre: Skipped 3 previous similar messages [14284.417812] Lustre: server umount lustre-MDT0000 complete [14289.230601] LustreError: 444409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786031137 with bad export cookie 8056304781548076924 [14289.246761] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14289.258116] LustreError: 444409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14289.783975] Lustre: server umount lustre-MDT0001 complete [14305.962696] Lustre: server umount lustre-OST0000 complete [14321.045718] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:46:07 (1786031167) [14331.449447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14332.074355] LustreError: 448415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14332.112807] LustreError: 448415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14332.196435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14337.508044] LustreError: 448416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14338.155290] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [14342.629674] LustreError: 448415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14347.752732] LustreError: 448416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14352.388522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14352.867484] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14353.358599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14361.066149] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [14374.442929] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14381.804478] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14403.127783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14403.542990] Lustre: Found index 0 for lustre-OST0000, updating log [14403.870246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14412.245603] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [14421.051402] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14428.491365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14439.907825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14439.928154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14439.963178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14441.079281] LustreError: 450248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14441.373980] Lustre: server umount lustre-OST0000 complete [14460.897519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14460.906482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14460.922959] Lustre: Skipped 1 previous similar message [14460.929977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14460.935489] Lustre: Skipped 1 previous similar message [14462.943954] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp [14462.954568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14465.184856] Lustre: server umount lustre-MDT0000 complete [14467.107957] LustreError: 448410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14467.141542] LustreError: 448410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14469.307545] LustreError: 448397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786031317 with bad export cookie 8056304781548078317 [14469.313100] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14469.324496] LustreError: 448397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14469.650628] Lustre: server umount lustre-MDT0001 complete [14482.075414] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:48:48 (1786031328) [14526.432380] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp [15111.019761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15111.331497] LustreError: 454857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15111.361764] LustreError: 454857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15111.445377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15115.552496] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15116.769826] LustreError: 454858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15121.901334] LustreError: 454857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15125.363849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15125.992300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15131.722795] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15143.341320] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15150.366268] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15165.536406] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp [15172.659625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15172.926195] Lustre: Found index 0 for lustre-OST0000, updating log [15173.151748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15180.068345] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15189.267380] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15192.710237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15192.732213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15192.737771] Lustre: Skipped 2 previous similar messages [15198.126524] Lustre: server umount lustre-OST0000 complete [15204.194413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15204.219692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15206.307107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15206.328047] Lustre: Skipped 4 previous similar messages [15209.735948] Lustre: server umount lustre-MDT0000 complete [15210.403311] LustreError: 454857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15210.420723] LustreError: 454857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15214.710703] LustreError: 456714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786032062 with bad export cookie 8056304781548079402 [15214.712614] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15214.722948] LustreError: 456714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15215.140289] Lustre: server umount lustre-MDT0001 complete [15225.951511] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp [15249.201529] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [15260.249308] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [15305.382586] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:02:31 (1786032151) [15327.868605] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [15331.294273] Key type lgssc unregistered [15331.614032] LNet: 461490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15331.621461] LNetError: 461490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15435.051015] LNet: Removed LNI 192.168.201.138@tcp [15436.178803] Key type .llcrypt unregistered [15436.184445] Key type ._llcrypt unregistered [15455.432193] Key type ._llcrypt registered [15455.434428] Key type .llcrypt registered [15455.578927] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [15457.300210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15457.376018] alg: No test for adler32 (adler32-zlib) [15458.607935] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [15458.888766] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [15460.631132] Key type lgssc registered [15462.354255] Lustre: Echo OBD driver; http://www.lustre.org/ [15469.509446] LNet: Added LNI 192.168.201.138@tcp100 [8/256/0/180] [15471.003030] LNet: Added LNI 192.168.201.138@tcp101 [8/256/0/180] [15472.691771] LNet: Added LNI 192.168.201.138@tcp102 [8/256/0/180] [15473.858712] LNet: Added LNI 192.168.201.138@tcp103 [8/256/0/180] [15475.537969] LNet: Added LNI 192.168.201.138@tcp104 [8/256/0/180] [15476.779130] LNet: Added LNI 192.168.201.138@tcp105 [8/256/0/180] [15477.893377] LNet: Added LNI 192.168.201.138@tcp106 [8/256/0/180] [15478.951642] LNet: Added LNI 192.168.201.138@tcp107 [8/256/0/180] [15479.952986] LNet: Added LNI 192.168.201.138@tcp108 [8/256/0/180] [15481.036212] LNet: Added LNI 192.168.201.138@tcp109 [8/256/0/180] [15482.322179] LNet: Added LNI 192.168.201.138@tcp110 [8/256/0/180] [15483.566359] LNet: Added LNI 192.168.201.138@tcp111 [8/256/0/180] [15484.784130] LNet: Added LNI 192.168.201.138@tcp112 [8/256/0/180] [15486.066477] LNet: Added LNI 192.168.201.138@tcp113 [8/256/0/180] [15487.177405] LNet: Added LNI 192.168.201.138@tcp114 [8/256/0/180] [15488.133583] LNet: Added LNI 192.168.201.138@tcp115 [8/256/0/180] [15489.278927] LNet: Added LNI 192.168.201.138@tcp116 [8/256/0/180] [15490.407050] LNet: Added LNI 192.168.201.138@tcp117 [8/256/0/180] [15491.669494] LNet: Added LNI 192.168.201.138@tcp118 [8/256/0/180] [15492.764250] LNet: Added LNI 192.168.201.138@tcp119 [8/256/0/180] [15493.998618] LNet: Added LNI 192.168.201.138@tcp120 [8/256/0/180] [15495.232517] LNet: Added LNI 192.168.201.138@tcp121 [8/256/0/180] [15496.427474] LNet: Added LNI 192.168.201.138@tcp122 [8/256/0/180] [15497.520672] LNet: Added LNI 192.168.201.138@tcp123 [8/256/0/180] [15498.576414] LNet: Added LNI 192.168.201.138@tcp124 [8/256/0/180] [15499.598902] LNet: Added LNI 192.168.201.138@tcp125 [8/256/0/180] [15500.682291] LNet: Added LNI 192.168.201.138@tcp126 [8/256/0/180] [15501.620262] LNet: Added LNI 192.168.201.138@tcp127 [8/256/0/180] [15502.609307] LNet: Added LNI 192.168.201.138@tcp128 [8/256/0/180] [15503.487821] LNet: Added LNI 192.168.201.138@tcp129 [8/256/0/180] [15504.542384] LNet: Added LNI 192.168.201.138@tcp130 [8/256/0/180] [15505.552589] LNet: Added LNI 192.168.201.138@tcp131 [8/256/0/180] [15506.569521] LNet: Added LNI 192.168.201.138@tcp132 [8/256/0/180] [15507.590510] LNet: Added LNI 192.168.201.138@tcp133 [8/256/0/180] [15508.560771] LNet: Added LNI 192.168.201.138@tcp134 [8/256/0/180] [15509.595283] LNet: Added LNI 192.168.201.138@tcp135 [8/256/0/180] [15517.858756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15517.883332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15519.026752] Lustre: 464523:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [15519.114754] LNetError: 461980:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [15519.153059] LNetError: Protocol error connecting to 192.168.201.138@tcp121 at host 192.168.201.138:988: is it running a compatible version of Lustre? [15519.180619] LNetError: Skipped 2 previous similar messages [15519.289885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15519.327867] LNet: 2 local NIs in recovery (showing 2): 192.168.201.138@tcp125, 192.168.201.138@tcp129 [15519.348767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15519.453771] Lustre: lustre-MDT0000: new disk, initializing [15519.542188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15519.555361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15520.224441] LNetError: 464548:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [15520.241860] LNetError: 464548:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [15520.261410] LNetError: Protocol error connecting to 192.168.201.138@tcp115 at host 192.168.201.138:988: is it running a compatible version of Lustre? [15520.303273] LNetError: Skipped 2 previous similar messages [15522.422098] LNetError: 464555:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [15522.435848] LNetError: 464555:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15522.454761] LNetError: Protocol error connecting to 192.168.201.138@tcp125 at host 192.168.201.138:988: is it running a compatible version of Lustre? [15522.462562] LNetError: Skipped 2 previous similar messages [15525.353156] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15526.513662] LNetError: 464550:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [15526.538551] LNetError: Protocol error connecting to 192.168.201.138@tcp125 at host 192.168.201.138:988: is it running a compatible version of Lustre? [15534.716587] LNetError: 464556:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [15534.736548] LNetError: Protocol error connecting to 192.168.201.138@tcp125 at host 192.168.201.138:988: is it running a compatible version of Lustre? [15537.314431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15537.582364] Lustre: 464538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15537.741958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15537.757697] Lustre: Skipped 1 previous similar message [15537.878281] Lustre: lustre-MDT0001: new disk, initializing [15537.914526] Lustre: 465342:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [15537.924120] Lustre: 465342:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15538.153630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15538.236183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15538.246572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15543.565715] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15546.444486] Lustre: 465707:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [15546.464111] Lustre: 465707:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15550.119794] LNetError: 461982:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [15550.130231] LNetError: Protocol error connecting to 192.168.201.138@tcp125 at host 192.168.201.138:988: is it running a compatible version of Lustre? [15555.071586] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15561.568605] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15566.435338] LNetError: 461981:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [15566.456375] LNetError: 461981:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [15566.461961] LNetError: Protocol error connecting to 192.168.201.138@tcp115 at host 192.168.201.138:988: is it running a compatible version of Lustre? [15566.474296] LNetError: Skipped 5 previous similar messages [15570.224905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15570.506460] Lustre: lustre-OST0000: new disk, initializing [15570.509679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15570.517642] Lustre: 466515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15570.626150] Lustre: 466520:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [15570.635855] Lustre: 466520:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15570.746110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15579.172379] Lustre: 466865:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.138@tcp: can't add NID 192.168.201.138@tcp130: rc = -75 [15579.193336] Lustre: 466865:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15579.211594] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15579.271925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15579.283386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15579.445531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15579.807258] LNet: 1 local NIs in recovery (showing 1): 192.168.201.138@tcp125 [15590.717371] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15599.202615] LNetError: 461981:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.138 [15599.225454] LNetError: 461981:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15599.231938] LNetError: Protocol error connecting to 192.168.201.138@tcp114 at host 192.168.201.138:988: is it running a compatible version of Lustre? [15599.241420] LNetError: Skipped 10 previous similar messages [15600.098452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15600.102582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15600.107283] LustreError: Skipped 1 previous similar message [15600.109490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15600.125825] Lustre: Skipped 1 previous similar message [15600.149124] Lustre: Skipped 1 previous similar message [15605.228492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15605.236859] Lustre: Skipped 1 previous similar message [15605.595874] Lustre: server umount lustre-OST0000 complete [15609.825123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15609.833886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15609.845128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15615.456970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15615.460073] Lustre: Skipped 2 previous similar messages [15616.127729] Lustre: server umount lustre-MDT0000 complete [15620.889546] LustreError: 464532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786032468 with bad export cookie 219030493772926891 [15620.890321] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15620.898250] LustreError: 464532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15621.347987] Lustre: server umount lustre-MDT0001 complete [15635.656622] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [15639.831737] Key type lgssc unregistered [15640.615436] LNet: 468432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15640.621346] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15640.636594] LNet: Removed LNI 192.168.201.138@tcp [15641.776817] LNet: Removed LNI 192.168.201.138@tcp100 [15641.794905] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15641.813157] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15641.826801] LNet: Removed LNI 192.168.201.138@tcp101 [15642.847374] LNet: Removed LNI 192.168.201.138@tcp102 [15642.858668] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15642.874086] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15642.880268] LNet: Removed LNI 192.168.201.138@tcp103 [15643.935598] LNet: Removed LNI 192.168.201.138@tcp104 [15644.960301] LNet: Removed LNI 192.168.201.138@tcp105 [15644.966340] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15644.994680] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15645.005605] LNet: Removed LNI 192.168.201.138@tcp106 [15646.047860] LNet: Removed LNI 192.168.201.138@tcp107 [15647.071712] LNet: Removed LNI 192.168.201.138@tcp108 [15648.095455] LNet: Removed LNI 192.168.201.138@tcp109 [15649.119540] LNet: Removed LNI 192.168.201.138@tcp110 [15649.124856] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15649.139846] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15649.151503] LNet: Removed LNI 192.168.201.138@tcp111 [15650.207458] LNet: Removed LNI 192.168.201.138@tcp112 [15651.231512] LNet: Removed LNI 192.168.201.138@tcp113 [15652.255353] LNet: Removed LNI 192.168.201.138@tcp114 [15653.279302] LNet: Removed LNI 192.168.201.138@tcp115 [15654.367163] LNet: Removed LNI 192.168.201.138@tcp116 [15655.391154] LNet: Removed LNI 192.168.201.138@tcp117 [15656.415232] LNet: Removed LNI 192.168.201.138@tcp118 [15657.439559] LNet: Removed LNI 192.168.201.138@tcp119 [15657.442050] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15657.443993] LNetError: 468432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15657.455460] LNet: Removed LNI 192.168.201.138@tcp120 [15658.463272] LNet: Removed LNI 192.168.201.138@tcp121 [15659.489407] LNet: Removed LNI 192.168.201.138@tcp122 [15660.511944] LNet: Removed LNI 192.168.201.138@tcp123 [15661.535202] LNet: Removed LNI 192.168.201.138@tcp124 [15661.542368] LNet: Removed LNI 192.168.201.138@tcp125 [15662.623273] LNet: Removed LNI 192.168.201.138@tcp126 [15663.647347] LNet: Removed LNI 192.168.201.138@tcp127 [15664.671210] LNet: Removed LNI 192.168.201.138@tcp128 [15665.695326] LNet: Removed LNI 192.168.201.138@tcp129 [15665.697287] LNet: Removed LNI 192.168.201.138@tcp130 [15665.704960] LNet: Removed LNI 192.168.201.138@tcp131 [15665.714334] LNet: Removed LNI 192.168.201.138@tcp132 [15665.718473] LNet: Removed LNI 192.168.201.138@tcp133 [15665.726203] LNet: Removed LNI 192.168.201.138@tcp134 [15665.742305] LNet: Removed LNI 192.168.201.138@tcp135 [15667.251118] Key type .llcrypt unregistered [15667.255120] Key type ._llcrypt unregistered [15695.428692] Key type ._llcrypt registered [15695.432418] Key type .llcrypt registered [15695.620354] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [15713.680385] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [15715.457394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15715.534343] alg: No test for adler32 (adler32-zlib) [15716.718913] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [15717.146633] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [15718.888410] Key type lgssc registered [15720.338982] Lustre: Echo OBD driver; http://www.lustre.org/ [15763.368821] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:10:10 (1786032610) [15785.067480] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [15788.976309] Key type lgssc unregistered [15789.372494] LNet: 472540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15789.387872] LNetError: 472540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15789.402965] LNet: Removed LNI 192.168.201.138@tcp [15790.418442] Key type .llcrypt unregistered [15790.420267] Key type ._llcrypt unregistered [15808.987617] Key type ._llcrypt registered [15808.994561] Key type .llcrypt registered [15809.142451] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [15810.151569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15810.258103] alg: No test for adler32 (adler32-zlib) [15811.747103] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [15812.019590] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [15813.655144] Key type lgssc registered [15815.147219] Lustre: Echo OBD driver; http://www.lustre.org/ [15827.588335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15827.673225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15829.040183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15829.089601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15829.188403] Lustre: lustre-MDT0000: new disk, initializing [15829.276776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15829.299586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15834.869952] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15847.592255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15847.739402] Lustre: 473754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15847.875658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15847.883086] Lustre: Skipped 1 previous similar message [15848.002809] Lustre: lustre-MDT0001: new disk, initializing [15848.183775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15848.244740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15848.259846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15853.624234] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15865.932769] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15873.467486] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15882.025093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15882.430864] Lustre: lustre-OST0000: new disk, initializing [15882.444674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15882.460417] Lustre: 475724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15882.675049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15887.967230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15887.992664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15888.074361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15890.839173] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [15903.567761] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15909.680901] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing lnet_if_list [15917.095839] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15917.138263] LNet: Added LNI 192.168.201.138@tcp10 [8/256/0/180] [15925.712863] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15925.746713] LNet: Added LNI 192.168.201.138@tcp11 [8/256/0/180] [15932.285668] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15932.341686] LNet: Added LNI 192.168.201.138@tcp12 [8/256/0/180] [15938.785499] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15938.826619] LNet: Added LNI 192.168.201.138@tcp13 [8/256/0/180] [15945.215348] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15945.258669] LNet: Added LNI 192.168.201.138@tcp14 [8/256/0/180] [15951.292355] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15951.323293] LNet: Added LNI 192.168.201.138@tcp15 [8/256/0/180] [15956.693213] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15956.722778] LNet: Added LNI 192.168.201.138@tcp16 [8/256/0/180] [15961.965237] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15961.988855] LNet: Added LNI 192.168.201.138@tcp17 [8/256/0/180] [15968.276175] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15968.309227] LNet: Added LNI 192.168.201.138@tcp18 [8/256/0/180] [15974.650536] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15974.696360] LNet: Added LNI 192.168.201.138@tcp19 [8/256/0/180] [15981.522066] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15981.580218] LNet: Added LNI 192.168.201.138@tcp20 [8/256/0/180] [15987.996899] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15988.031838] LNet: Added LNI 192.168.201.138@tcp21 [8/256/0/180] [15994.257936] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15994.342515] LNet: Added LNI 192.168.201.138@tcp22 [8/256/0/180] [16001.541881] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16001.564972] LNet: Added LNI 192.168.201.138@tcp23 [8/256/0/180] [16008.425654] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16008.458957] LNet: Added LNI 192.168.201.138@tcp24 [8/256/0/180] [16014.284440] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16014.304578] LNet: Added LNI 192.168.201.138@tcp25 [8/256/0/180] [16021.159219] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16021.192787] LNet: Added LNI 192.168.201.138@tcp26 [8/256/0/180] [16027.296235] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16027.322769] LNet: Added LNI 192.168.201.138@tcp27 [8/256/0/180] [16034.570688] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16034.608944] LNet: Added LNI 192.168.201.138@tcp28 [8/256/0/180] [16044.118721] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16044.178299] LNet: Added LNI 192.168.201.138@tcp29 [8/256/0/180] [16052.873911] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16052.917661] LNet: Added LNI 192.168.201.138@tcp30 [8/256/0/180] [16061.135865] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16061.174927] LNet: Added LNI 192.168.201.138@tcp31 [8/256/0/180] [16068.996165] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16069.037479] LNet: Added LNI 192.168.201.138@tcp32 [8/256/0/180] [16075.945662] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16075.968387] LNet: Added LNI 192.168.201.138@tcp33 [8/256/0/180] [16083.234056] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16083.257799] LNet: Added LNI 192.168.201.138@tcp34 [8/256/0/180] [16092.409275] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16092.459635] LNet: Added LNI 192.168.201.138@tcp35 [8/256/0/180] [16102.057303] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16102.100250] LNet: Added LNI 192.168.201.138@tcp36 [8/256/0/180] [16112.024466] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16112.044370] LNet: Added LNI 192.168.201.138@tcp37 [8/256/0/180] [16120.031821] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16120.061575] LNet: Added LNI 192.168.201.138@tcp38 [8/256/0/180] [16127.693789] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16127.714775] LNet: Added LNI 192.168.201.138@tcp39 [8/256/0/180] [16137.033240] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16137.100483] LNet: Added LNI 192.168.201.138@tcp40 [8/256/0/180] [16146.291322] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16146.323738] LNet: Added LNI 192.168.201.138@tcp41 [8/256/0/180] [16154.550823] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16154.588862] LNet: Added LNI 192.168.201.138@tcp42 [8/256/0/180] [16164.729740] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16164.750631] LNet: Added LNI 192.168.201.138@tcp43 [8/256/0/180] [16173.645386] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16173.659596] LNet: Added LNI 192.168.201.138@tcp44 [8/256/0/180] [16182.669447] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16182.722484] LNet: Added LNI 192.168.201.138@tcp45 [8/256/0/180] [16193.763783] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16193.793747] LNet: Added LNI 192.168.201.138@tcp46 [8/256/0/180] [16205.858228] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16205.901836] LNet: Added LNI 192.168.201.138@tcp47 [8/256/0/180] [16216.698569] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16216.748955] LNet: Added LNI 192.168.201.138@tcp48 [8/256/0/180] [16226.660035] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16226.701405] LNet: Added LNI 192.168.201.138@tcp49 [8/256/0/180] [16235.194382] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16235.227255] LNet: Added LNI 192.168.201.138@tcp50 [8/256/0/180] [16257.004136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16257.007209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16257.027804] Lustre: Skipped 1 previous similar message [16257.051110] Lustre: Skipped 2 previous similar messages [16262.114214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16262.124404] Lustre: Skipped 2 previous similar messages [16267.236477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16267.249856] Lustre: Skipped 2 previous similar messages [16269.279169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16269.635672] Lustre: server umount lustre-MDT0000 complete [16274.128174] LustreError: 473747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786033122 with bad export cookie 12099526905730106716 [16274.128749] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16274.151929] LustreError: 473747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16274.757696] Lustre: server umount lustre-MDT0001 complete [16293.860865] Lustre: 473026:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786033125/real 1786033125] req@0000000001460472 x1872790981640192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786033141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16293.891688] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16293.906196] Lustre: Skipped 1 previous similar message [16295.967094] Lustre: 473026:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786033127/real 1786033127] req@00000000d3c25488 x1872790981640320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786033143 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16296.199237] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [16298.978986] Lustre: 473028:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786033130/real 1786033130] req@00000000ff896b92 x1872790981640448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786033146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16301.024491] Lustre: 473029:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786033132/real 1786033132] req@00000000007f8015 x1872790981640704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786033148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16305.121222] Lustre: 473026:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786033137/real 1786033137] req@00000000b5bca4b2 x1872790981640832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786033153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16309.547209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16319.944942] LustreError: 475717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16319.956473] LustreError: 475717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16319.998356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16324.006952] LustreError: 475718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16325.180962] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16327.020369] LustreError: 475717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16328.059396] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16333.294220] LustreError: 475718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16333.330294] LustreError: 475718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16335.369359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16335.798884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16338.884147] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16340.761413] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16349.170051] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16356.018505] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16385.005420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16385.015972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16385.037434] Lustre: Skipped 1 previous similar message [16385.052526] Lustre: Skipped 1 previous similar message [16388.764477] Lustre: server umount lustre-OST0000 complete [16395.233116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16395.257700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16399.560996] Lustre: server umount lustre-MDT0000 complete [16400.357685] LustreError: 499977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16400.374695] LustreError: 499977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16403.137418] LustreError: 473748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786033251 with bad export cookie 12099526905730109425 [16403.138850] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16403.148235] LustreError: 473748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16403.524779] Lustre: server umount lustre-MDT0001 complete [16414.810744] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [16417.733972] Key type lgssc unregistered [16418.123484] LNet: 502777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16418.134582] LNetError: 502777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16419.168077] LNet: Removed LNI 192.168.201.138@tcp [16419.172177] LNetError: 502777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16419.181059] LNet: Removed LNI 192.168.201.138@tcp10 [16419.182468] LNet: Removed LNI 192.168.201.138@tcp11 [16419.192475] LNet: Removed LNI 192.168.201.138@tcp12 [16419.194787] LNet: Removed LNI 192.168.201.138@tcp13 [16419.197742] LNet: Removed LNI 192.168.201.138@tcp14 [16419.201225] LNet: Removed LNI 192.168.201.138@tcp15 [16419.203310] LNet: Removed LNI 192.168.201.138@tcp16 [16419.206274] LNet: Removed LNI 192.168.201.138@tcp17 [16419.209203] LNet: Removed LNI 192.168.201.138@tcp18 [16419.212130] LNet: Removed LNI 192.168.201.138@tcp19 [16419.215035] LNet: Removed LNI 192.168.201.138@tcp20 [16419.221861] LNet: Removed LNI 192.168.201.138@tcp21 [16419.229149] LNet: Removed LNI 192.168.201.138@tcp22 [16419.232137] LNet: Removed LNI 192.168.201.138@tcp23 [16419.234315] LNet: Removed LNI 192.168.201.138@tcp24 [16419.236210] LNet: Removed LNI 192.168.201.138@tcp25 [16419.239196] LNet: Removed LNI 192.168.201.138@tcp26 [16419.241729] LNet: Removed LNI 192.168.201.138@tcp27 [16419.248500] LNet: Removed LNI 192.168.201.138@tcp28 [16419.253395] LNet: Removed LNI 192.168.201.138@tcp29 [16419.257071] LNet: Removed LNI 192.168.201.138@tcp30 [16419.260104] LNet: Removed LNI 192.168.201.138@tcp31 [16419.262978] LNet: Removed LNI 192.168.201.138@tcp32 [16419.265582] LNet: Removed LNI 192.168.201.138@tcp33 [16419.268239] LNet: Removed LNI 192.168.201.138@tcp34 [16419.274173] LNet: Removed LNI 192.168.201.138@tcp35 [16419.276847] LNet: Removed LNI 192.168.201.138@tcp36 [16419.279227] LNet: Removed LNI 192.168.201.138@tcp37 [16419.281935] LNet: Removed LNI 192.168.201.138@tcp38 [16419.284561] LNet: Removed LNI 192.168.201.138@tcp39 [16419.287345] LNet: Removed LNI 192.168.201.138@tcp40 [16419.290698] LNet: Removed LNI 192.168.201.138@tcp41 [16419.293099] LNet: Removed LNI 192.168.201.138@tcp42 [16419.295992] LNet: Removed LNI 192.168.201.138@tcp43 [16419.298403] LNet: Removed LNI 192.168.201.138@tcp44 [16419.301145] LNet: Removed LNI 192.168.201.138@tcp45 [16419.305179] LNet: Removed LNI 192.168.201.138@tcp46 [16419.307984] LNet: Removed LNI 192.168.201.138@tcp47 [16419.310292] LNet: Removed LNI 192.168.201.138@tcp48 [16419.319538] LNet: Removed LNI 192.168.201.138@tcp49 [16419.330905] LNet: Removed LNI 192.168.201.138@tcp50 [16420.412455] Key type .llcrypt unregistered [16420.414407] Key type ._llcrypt unregistered [16442.388469] Key type ._llcrypt registered [16442.391679] Key type .llcrypt registered [16442.440772] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [16460.844224] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [16462.409969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16462.517702] alg: No test for adler32 (adler32-zlib) [16463.759433] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [16463.958266] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [16465.623623] Key type lgssc registered [16466.575895] Lustre: Echo OBD driver; http://www.lustre.org/ [16510.777712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16510.832609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16512.196467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16512.237020] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16512.294495] Lustre: lustre-MDT0000: new disk, initializing [16512.353789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16512.369732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16517.044282] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16529.330167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16529.413743] Lustre: 505978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16529.490088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16529.495274] Lustre: Skipped 1 previous similar message [16529.621270] Lustre: lustre-MDT0001: new disk, initializing [16529.692512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16529.718414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16529.724275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16534.806201] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16546.760800] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16553.701766] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16562.299325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16562.554283] Lustre: *** cfs_fail_loc=90f, val=0*** [16562.613979] Lustre: lustre-OST0000: new disk, initializing [16562.617315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16562.624153] Lustre: 507996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16562.713729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16568.395579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16568.404708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16568.569240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16569.150742] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16580.076680] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16589.285208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16589.291050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16589.299969] LustreError: Skipped 1 previous similar message [16589.301996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16589.317773] Lustre: Skipped 1 previous similar message [16589.345350] Lustre: Skipped 1 previous similar message [16594.160868] Lustre: server umount lustre-OST0000 complete [16599.290309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16599.314067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16599.320298] Lustre: Skipped 1 previous similar message [16604.641413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16604.647600] Lustre: Skipped 1 previous similar message [16605.641404] Lustre: server umount lustre-MDT0000 complete [16610.310672] LustreError: 505971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786033458 with bad export cookie 9160330819925218692 [16610.330458] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16610.333526] LustreError: 505971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16610.733606] Lustre: server umount lustre-MDT0001 complete [16622.663546] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [16625.977370] Key type lgssc unregistered [16626.265822] LNet: 509905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16626.271095] LNetError: 509905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16627.304271] LNet: Removed LNI 192.168.201.138@tcp [16628.128918] Key type .llcrypt unregistered [16628.134390] Key type ._llcrypt unregistered [16649.838448] Key type ._llcrypt registered [16649.846034] Key type .llcrypt registered [16649.978837] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [16650.995909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16651.063841] alg: No test for adler32 (adler32-zlib) [16652.082854] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [16652.382847] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [16654.079328] Key type lgssc registered [16655.271256] Lustre: Echo OBD driver; http://www.lustre.org/ [16663.796622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16663.810740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16665.326886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16671.234527] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16675.823796] LustreError: 511176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16675.850764] LustreError: 511176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16680.781602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16680.929558] LustreError: 511177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16681.427283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16687.240366] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16696.846846] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16703.721455] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16711.504651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16711.956047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16719.551797] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16728.803374] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16750.701138] loop0: detected capacity change from 0 to 204800000 [16750.906722] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16750.920504] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16750.965093] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16758.358619] loop0: detected capacity change from 0 to 204800000 [16758.539571] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16758.546677] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16758.558296] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16764.820853] loop0: detected capacity change from 0 to 204800000 [16771.356674] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16771.510857] Lustre: fs15246-OST0000: new disk, initializing [16771.515504] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16771.521768] Lustre: 514794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16771.606730] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16777.573505] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16786.941273] loop1: detected capacity change from 0 to 204800000 [16793.049122] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16793.144812] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16793.182615] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16793.246836] Lustre: fs15246-MDT0000: new disk, initializing [16793.418324] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16793.447454] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16793.532668] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16794.594238] Lustre: 515917:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a791bc31 x1872791863348352/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786033658 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16798.121634] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [16803.681489] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16803.692286] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16803.740411] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16823.283915] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16823.307354] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16828.386665] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16828.975759] Lustre: server umount fs15246-MDT0000 complete [16833.512309] LustreError: 511171:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16838.635415] LustreError: 511171:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16840.961900] Lustre: server umount fs15246-OST0000 complete [16858.593526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16858.598706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16858.613769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16864.227192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16864.241025] Lustre: Skipped 3 previous similar messages [16869.345643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16869.364348] Lustre: Skipped 1 previous similar message [16870.880068] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16871.115223] Lustre: server umount lustre-OST0000 complete [16879.587466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16879.604245] Lustre: Skipped 1 previous similar message [16879.611219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16879.617152] Lustre: Skipped 1 previous similar message [16883.096995] Lustre: server umount lustre-MDT0000 complete [16884.709747] LustreError: 511171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16884.763759] LustreError: 511171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16887.334301] LustreError: 517973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786033735 with bad export cookie 9493556052508385446 [16887.339316] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16887.355483] LustreError: 517973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16887.999708] Lustre: server umount lustre-MDT0001 complete [16902.273842] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [16906.036250] Key type lgssc unregistered [16906.247602] LNet: 519015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16906.281253] LNetError: 519015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16906.299364] LNet: Removed LNI 192.168.201.138@tcp [16907.543988] Key type .llcrypt unregistered [16907.546035] Key type ._llcrypt unregistered [16933.279166] Key type ._llcrypt registered [16933.280297] Key type .llcrypt registered [16933.375368] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [16934.443628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16934.467495] alg: No test for adler32 (adler32-zlib) [16935.854430] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [16936.192730] LNet: Added LNI 192.168.201.138@tcp [8/256/0/180] [16937.879201] Key type lgssc registered [16939.106693] Lustre: Echo OBD driver; http://www.lustre.org/ [16983.202632] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [16994.028120] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [17039.017797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17039.138496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17040.722909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17040.790592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17040.876738] Lustre: lustre-MDT0000: new disk, initializing [17041.020368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17041.056950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17045.646116] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [17057.126177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17057.274279] Lustre: 523490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17057.322682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17057.325622] Lustre: Skipped 1 previous similar message [17057.393181] Lustre: lustre-MDT0001: new disk, initializing [17057.510880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17057.590982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17057.599740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17063.984303] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [17078.895270] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17086.502722] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17095.644480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17096.126709] Lustre: lustre-OST0000: new disk, initializing [17096.133100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17096.139075] Lustre: 525459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17096.254904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17104.687545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17104.712525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17104.971173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17107.386359] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_default_debug -1 all [17119.481374] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17125.786401] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17126.164191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17132.196588] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17132.502841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17140.704880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17140.716622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17140.736453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17140.743077] Lustre: Skipped 1 previous similar message [17140.759127] Lustre: Skipped 1 previous similar message [17142.226693] Lustre: server umount lustre-OST0000 complete [17149.927303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17149.933510] LustreError: Skipped 1 previous similar message [17149.936849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17149.947847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17150.945355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17150.962940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17150.971139] Lustre: Skipped 1 previous similar message [17152.904355] Lustre: server umount lustre-MDT0000 complete [17157.355607] LustreError: 523482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786034005 with bad export cookie 16521749677079944223 [17157.357246] LustreError: MGC192.168.201.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17157.364398] LustreError: 523482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17157.769554] Lustre: server umount lustre-MDT0001 complete [17186.428577] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing set_hostid [17201.588251] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing load_modules_local [17239.744860] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16994 sec ========= 12:34:46 (1786034086) [17241.439428] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:34:48 (1786034088) === [17245.477476] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:34:51 (1786034091) === [17288.554697] Lustre: DEBUG MARKER: oleg138-server.virtnet: executing unload_modules_local [17291.648447] Key type lgssc unregistered [17291.919511] LNet: 532127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17291.934809] LNetError: 532127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17291.953761] LNet: Removed LNI 192.168.201.138@tcp [17292.896564] Key type .llcrypt unregistered [17292.903443] Key type ._llcrypt unregistered