[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493311889 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007507] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010152] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012060] SELinux: Initializing. [ 0.013089] *** VALIDATE selinux *** [ 0.021861] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026850] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028025] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031054] *** VALIDATE tmpfs *** [ 0.033316] *** VALIDATE proc *** [ 0.034306] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.036349] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037181] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043702] debug: unmapping init [mem 0xffffffff9ea59000-0xffffffff9ea60fff] [ 0.045213] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046748] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.047921] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.048915] ... value mask: 0000ffffffffffff [ 0.049020] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052353] rcu: Hierarchical SRCU implementation. [ 0.054498] smp: Bringing up secondary CPUs ... [ 0.055600] x86: Booting SMP configuration: [ 0.056018] .... node #0, CPUs: #1 #2 #3 [ 0.061018] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064042] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.154315] node 0 deferred pages initialised in 87ms [ 0.157119] devtmpfs: initialized [ 0.158294] x86/mm: Memory block size: 128MB [ 0.161907] gcov: version magic: 0x41383552 [ 0.164166] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166259] pinctrl core: initialized pinctrl subsystem [ 0.167180] [ 0.167716] ************************************************************* [ 0.168018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169016] ** ** [ 0.170019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171013] ** ** [ 0.172016] ** This means that this kernel is built to expose internal ** [ 0.173015] ** IOMMU data structures, which may compromise security on ** [ 0.174012] ** your system. ** [ 0.175014] ** ** [ 0.176013] ** If you see this message and you are not debugging the ** [ 0.177015] ** kernel, report this immediately to your vendor! ** [ 0.178016] ** ** [ 0.179013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180013] ************************************************************* [ 0.181785] NET: Registered protocol family 16 [ 0.182460] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185487] cpuidle: using governor menu [ 0.188508] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189461] PCI: Using configuration type 1 for base access [ 0.190125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.200100] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204092] cryptd: max_cpu_qlen set to 1000 [ 0.207178] ACPI: Added _OSI(Module Device) [ 0.208013] ACPI: Added _OSI(Processor Device) [ 0.210018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.212018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218870] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225623] ACPI: Interpreter enabled [ 0.226066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227016] ACPI: Using IOAPIC for interrupt routing [ 0.228122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229462] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.240040] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245445] acpiphp: Slot [2] registered [ 0.246119] acpiphp: Slot [5] registered [ 0.247109] acpiphp: Slot [6] registered [ 0.248129] acpiphp: Slot [7] registered [ 0.249125] acpiphp: Slot [8] registered [ 0.250111] acpiphp: Slot [9] registered [ 0.251174] acpiphp: Slot [10] registered [ 0.252139] acpiphp: Slot [3] registered [ 0.253099] acpiphp: Slot [4] registered [ 0.254144] acpiphp: Slot [11] registered [ 0.255123] acpiphp: Slot [12] registered [ 0.256111] acpiphp: Slot [13] registered [ 0.257172] acpiphp: Slot [14] registered [ 0.258121] acpiphp: Slot [15] registered [ 0.259103] acpiphp: Slot [16] registered [ 0.260116] acpiphp: Slot [17] registered [ 0.261119] acpiphp: Slot [18] registered [ 0.262110] acpiphp: Slot [19] registered [ 0.263155] acpiphp: Slot [20] registered [ 0.264096] acpiphp: Slot [21] registered [ 0.265149] acpiphp: Slot [22] registered [ 0.266101] acpiphp: Slot [23] registered [ 0.267128] acpiphp: Slot [24] registered [ 0.268122] acpiphp: Slot [25] registered [ 0.269126] acpiphp: Slot [26] registered [ 0.270104] acpiphp: Slot [27] registered [ 0.271120] acpiphp: Slot [28] registered [ 0.272090] acpiphp: Slot [29] registered [ 0.273099] acpiphp: Slot [30] registered [ 0.274102] acpiphp: Slot [31] registered [ 0.275073] PCI host bridge to bus 0000:00 [ 0.276032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.278040] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.281036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.282039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.283207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.285013] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287506] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.293615] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.296061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.297019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.298035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.299021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.300606] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.302046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.303746] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.309019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.324992] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.330015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.337105] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.347016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.353022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.386021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.401018] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.414020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.430031] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.486940] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.495026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.503021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545570] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.555017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.560020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.587018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.599476] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.609017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.614019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.647645] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.667020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.684017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.705016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.717315] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.719333] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.721215] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.722271] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.724428] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.728209] iommu: Default domain type: Passthrough [ 0.730303] SCSI subsystem initialized [ 0.731097] ACPI: bus type USB registered [ 0.732093] usbcore: registered new interface driver usbfs [ 0.733055] usbcore: registered new interface driver hub [ 0.734062] usbcore: registered new device driver usb [ 0.735118] pps_core: LinuxPPS API ver. 1 registered [ 0.736008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.738049] PTP clock support registered [ 0.739126] EDAC MC: Ver: 3.0.0 [ 0.740657] PCI: Using ACPI for IRQ routing [ 0.741502] NetLabel: Initializing [ 0.742000] NetLabel: domain hash size = 128 [ 0.744023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.746066] NetLabel: unlabeled traffic allowed by default [ 0.748082] vgaarb: loaded [ 0.750183] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.756000] clocksource: Switched to clocksource kvm-clock [ 0.890902] VFS: Disk quotas dquot_6.6.0 [ 0.892700] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.895710] *** VALIDATE ramfs *** [ 0.897391] *** VALIDATE hugetlbfs *** [ 0.899199] pnp: PnP ACPI init [ 0.902137] pnp: PnP ACPI: found 6 devices [ 0.937601] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.940879] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.943118] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.945121] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.947355] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.949669] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.952582] NET: Registered protocol family 2 [ 0.955069] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.960210] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.963406] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.968282] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.971555] TCP: Hash tables configured (established 65536 bind 65536) [ 0.974296] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.977246] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.979878] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982400] NET: Registered protocol family 1 [ 0.985340] RPC: Registered named UNIX socket transport module. [ 0.987281] RPC: Registered udp transport module. [ 0.988596] RPC: Registered tcp transport module. [ 0.989911] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.991724] NET: Registered protocol family 44 [ 0.993065] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.994758] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.996487] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.998439] PCI: CLS 0 bytes, default 64 [ 0.999638] Unpacking initramfs... [ 2.462177] debug: unmapping init [mem 0xffff957a3cc54000-0xffff957a3ffbffff] [ 2.466234] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.468460] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.471505] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.008528] Initialise system trusted keyrings [ 3.010302] Key type blacklist registered [ 3.012518] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.022133] zbud: loaded [ 3.025240] *** VALIDATE nfs *** [ 3.026601] *** VALIDATE nfs4 *** [ 3.028303] pstore: using deflate compression [ 3.032429] Platform Keyring initialized [ 3.139710] NET: Registered protocol family 38 [ 3.143231] Key type asymmetric registered [ 3.145240] Asymmetric key parser 'x509' registered [ 3.146935] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.150205] io scheduler mq-deadline registered [ 3.151833] io scheduler kyber registered [ 3.153581] io scheduler bfq registered [ 3.155381] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.158358] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.161245] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.164294] ACPI: Power Button [PWRF] [ 3.170088] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.177391] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.190843] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.198498] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.215594] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.243403] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.273498] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.278285] Non-volatile memory driver v1.3 [ 3.279697] Linux agpgart interface v0.103 [ 3.311071] virtio_blk virtio1: [vda] 150648 512-byte logical blocks (77.1 MB/73.6 MiB) [ 3.314080] vda: detected capacity change from 0 to 77131776 [ 3.338720] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.341780] vdb: detected capacity change from 0 to 1073741824 [ 3.356362] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.359261] vdc: detected capacity change from 0 to 2621440000 [ 3.371824] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.374753] vdd: detected capacity change from 0 to 2621440000 [ 3.387834] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.390767] vde: detected capacity change from 0 to 4294967296 [ 3.406319] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.409262] vdf: detected capacity change from 0 to 4294967296 [ 3.422247] libphy: Fixed MDIO Bus: probed [ 3.430775] usbcore: registered new interface driver usbserial_generic [ 3.433605] usbserial: USB Serial support registered for generic [ 3.436335] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.441298] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.443300] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.445994] mousedev: PS/2 mouse device common for all mice [ 3.448932] rtc_cmos 00:05: RTC can wake from S4 [ 3.451525] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.455471] rtc_cmos 00:05: registered as rtc0 [ 3.458320] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.460262] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.463177] intel_pstate: CPU model not supported [ 3.468095] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.474310] hid: raw HID events driver (C) Jiri Kosina [ 3.482178] usbcore: registered new interface driver usbhid [ 3.484583] usbhid: USB HID core driver [ 3.486411] drop_monitor: Initializing network drop monitor service [ 3.489340] Initializing XFRM netlink socket [ 3.492271] NET: Registered protocol family 10 [ 3.495937] Segment Routing with IPv6 [ 3.497636] NET: Registered protocol family 17 [ 3.500116] mpls_gso: MPLS GSO support [ 3.506685] RAS: Correctable Errors collector initialized. [ 3.509023] AVX version of gcm_enc/dec engaged. [ 3.510688] AES CTR mode by8 optimization enabled [ 3.595410] sched_clock: Marking stable (3595352456, 0)->(4603152166, -1007799710) [ 3.598783] registered taskstats version 1 [ 3.600206] Loading compiled-in X.509 certificates [ 3.601275] zswap: loaded using pool lzo/zbud [ 3.619467] Key type big_key registered [ 3.630293] Key type encrypted registered [ 3.631496] ima: No TPM chip found, activating TPM-bypass! [ 3.632946] ima: Allocated hash algorithm: sha1 [ 3.634347] ima: No architecture policies found [ 3.635505] evm: Initialising EVM extended attributes: [ 3.636835] evm: security.selinux [ 3.637781] evm: security.ima [ 3.638547] evm: security.capability [ 3.639583] evm: HMAC attrs: 0x1 [ 3.641478] rtc_cmos 00:05: setting system clock to 2026-09-30 02:12:39 UTC (1790734359) [ 3.646699] debug: unmapping init [mem 0xffffffff9fa03000-0xffffffff9fbfffff] [ 3.649049] debug: unmapping init [mem 0xffffffff9e782000-0xffffffff9ea58fff] [ 3.658114] Write protecting the kernel read-only data: 28672k [ 3.661660] debug: unmapping init [mem 0xffffffff9ce03000-0xffffffff9cffffff] [ 3.663773] debug: unmapping init [mem 0xffffffff9d714000-0xffffffff9d7fffff] [ 3.694426] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.700874] systemd[1]: Detected virtualization kvm. [ 3.702748] systemd[1]: Detected architecture x86-64. [ 3.704122] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.728514] systemd[1]: No hostname configured. [ 3.729741] systemd[1]: Set hostname to . [ 3.731202] random: systemd: uninitialized urandom read (16 bytes read) [ 3.732881] systemd[1]: Initializing machine ID from random generator. [ 3.859143] random: systemd: uninitialized urandom read (16 bytes read) [ 3.861577] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.865231] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871042] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.876513] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.381208] device-mapper: uevent: version 1.0.3 [ 4.382858] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.132801] virtio_net virtio0 ens2: renamed from eth0 [ 5.143828] random: fast init done [ 5.173389] scsi host0: ata_piix [ 5.181309] scsi host1: ata_piix [ 5.182727] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.184980] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.470595] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.148705] random: crng init done [ 10.150289] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.426755] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.447537] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.700540] SELinux: Disabled at runtime. [ 11.768839] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.778667] systemd[1]: Detected virtualization kvm. [ 11.780605] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.264916] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.268128] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.272709] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.276460] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.279598] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.288652] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.301412] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.470429] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.721439] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.015098] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.024798] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.109523] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.133409] EDAC sbridge: Ver: 1.1.2 [ 14.465150] Key type dns_resolver registered [ 14.742716] NFS: Registering the id_resolver key type [ 14.744896] Key type id_resolver registered [ 14.746498] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg135-server login: [ 47.391424] hrtimer: interrupt took 3532887 ns [ 68.847652] libcfs: loading out-of-tree module taints kernel. [ 68.906496] Key type ._llcrypt registered [ 68.908881] Key type .llcrypt registered [ 69.034852] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 91.375208] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 93.209988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.229341] alg: No test for adler32 (adler32-zlib) [ 95.112376] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 96.341265] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 98.224038] Key type lgssc registered [ 100.265887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.400383] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.426587] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 177.013610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 177.078299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.436660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.480276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.633642] Lustre: lustre-MDT0000: new disk, initializing [ 178.744933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.758391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.141245] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.348383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.634283] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.731328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.737212] Lustre: Skipped 1 previous similar message [ 199.908921] Lustre: lustre-MDT0001: new disk, initializing [ 200.035589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.083173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.102703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.731805] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.134647] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.840442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.249745] Lustre: lustre-OST0000: new disk, initializing [ 223.261476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.273884] Lustre: 8454:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.415395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.434464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.442655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.609417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.111418] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.549439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.702534] Lustre: lustre-OST0001: new disk, initializing [ 247.705882] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 247.709789] Lustre: 9526:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.770107] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 254.854058] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.057458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.079341] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.202886] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 267.286916] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.406463] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 281.656787] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing check_logdir /tmp/testlogs/ [ 287.322527] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing yml_node [ 291.508946] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 294.170930] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 296.984453] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 298.988884] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 29 22:17:33 EDT 2026 [ 316.720186] Lustre: DEBUG MARKER: only running tests: 33-75 [ 318.365624] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 320.244918] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.399803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.411064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.426192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.432114] Lustre: Skipped 3 previous similar messages [ 338.447662] Lustre: Skipped 3 previous similar messages [ 342.636442] Lustre: server umount lustre-MDT0000 complete [ 346.481682] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790734702 with bad export cookie 13138189637293687510 [ 346.491400] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 346.502271] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.639929] LustreError: 10102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.659687] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.663740] LustreError: 10102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 348.679985] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.685983] Lustre: Skipped 1 previous similar message [ 348.719489] Lustre: Skipped 1 previous similar message [ 351.649410] LustreError: 10102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.649594] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.678964] LustreError: 10102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 351.690690] Lustre: Skipped 1 previous similar message [ 353.049677] Lustre: server umount lustre-MDT0001 complete [ 357.438597] Lustre: server umount lustre-OST0000 complete [ 362.089336] Lustre: server umount lustre-OST0001 complete [ 370.874456] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 382.253713] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 421.818958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.233467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.293297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.414573] Lustre: lustre-MDT0000: new disk, initializing [ 422.616261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.644296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.771134] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 439.484556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.624789] Lustre: 15125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 439.696784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 439.700654] Lustre: Skipped 1 previous similar message [ 439.791693] Lustre: lustre-MDT0001: new disk, initializing [ 439.931494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 439.969880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 439.998077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.369780] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 455.837911] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 461.988602] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 468.861368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 469.108398] Lustre: lustre-OST0000: new disk, initializing [ 469.111458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 469.116243] Lustre: 17096:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 469.186382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.396281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.408692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.552806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 475.319280] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 486.581586] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 493.261895] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 493.646106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 498.634788] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.904556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 501.218947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.225346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.229353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.338587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.347902] Lustre: Skipped 2 previous similar messages [ 507.426818] Lustre: server umount lustre-OST0000 complete [ 511.458124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.467948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 511.473521] Lustre: Skipped 1 previous similar message [ 516.576986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.579794] Lustre: Skipped 3 previous similar messages [ 517.341428] Lustre: server umount lustre-MDT0000 complete [ 520.534263] LustreError: 15120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790734876 with bad export cookie 13138189637293689036 [ 520.537658] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.542669] LustreError: 15120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 520.779408] Lustre: server umount lustre-MDT0001 complete [ 524.596082] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:21:19 (1790734879) [ 527.294797] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 528.950340] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:21:23 (1790734883) [ 536.422515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.117645] LustreError: 19505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.219269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.202278] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 542.689600] LustreError: 19506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.815277] LustreError: 19505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.643796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.685156] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 564.524638] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.687926] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 580.374035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 589.647150] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 598.406847] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 607.724862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 607.729295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.735892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.743246] Lustre: Skipped 2 previous similar messages [ 607.748592] Lustre: Skipped 1 previous similar message [ 611.102861] Lustre: server umount lustre-OST0000 complete [ 617.956118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.968415] Lustre: server umount lustre-MDT0000 complete [ 623.075336] LustreError: 19505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.098413] LustreError: 19505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 625.187391] LustreError: 20240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790734981 with bad export cookie 13138189637293689918 [ 625.188157] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.216735] LustreError: 20240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 625.493962] Lustre: server umount lustre-MDT0001 complete [ 636.378131] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 639.275801] Key type lgssc unregistered [ 639.577296] LNet: 23039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.586798] LNetError: 23039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.619464] LNet: Removed LNI 192.168.201.135@tcp [ 641.608352] Key type .llcrypt unregistered [ 641.612781] Key type ._llcrypt unregistered [ 670.409577] Key type ._llcrypt registered [ 670.411418] Key type .llcrypt registered [ 670.479523] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 671.781802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 671.897236] alg: No test for adler32 (adler32-zlib) [ 672.987687] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 673.312774] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 674.959352] Key type lgssc registered [ 675.984586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 686.793439] loop: module loaded [ 686.874100] loop0: detected capacity change from 0 to 204800000 [ 687.052887] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.063133] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.350808] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.638221] loop0: detected capacity change from 0 to 204800000 [ 692.875732] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.897948] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.907679] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.828430] loop0: detected capacity change from 0 to 204800000 [ 705.638252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 705.650991] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.076734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 707.147832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 707.245358] Lustre: lustre-MDT0000: new disk, initializing [ 707.443895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.467521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 713.012592] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 722.271388] loop1: detected capacity change from 0 to 204800000 [ 728.079881] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 728.330620] Lustre: lustre-OST07c6: new disk, initializing [ 728.333239] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 728.335743] Lustre: Skipped 1 previous similar message [ 728.339709] Lustre: 26245:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 728.390630] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 734.748519] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 738.404280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 738.412861] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 738.490797] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 744.553961] Lustre: 25234:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 744.572543] Lustre: 25234:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 744.584615] Lustre: 25234:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 744.596402] Lustre: 25234:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 744.601692] Lustre: 25234:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 744.608949] Lustre: 25234:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 764.470995] Lustre: 26760:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 764.478101] Lustre: 26760:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 764.485470] Lustre: 26760:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.490257] Lustre: 26760:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 764.498403] Lustre: 26760:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 764.503653] Lustre: 26760:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.892096] Lustre: 25233:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 769.899599] Lustre: 25233:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 769.905159] Lustre: 25233:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 769.918395] Lustre: 25233:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.926855] Lustre: 25233:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.934096] Lustre: 25233:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.944664] Lustre: 25233:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 769.956287] Lustre: 25233:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.974991] Lustre: 25233:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 769.990054] Lustre: 25233:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.001191] Lustre: 25233:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.006071] Lustre: 25233:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.196963] Lustre: 25232:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 773.201946] Lustre: 25232:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 773.206209] Lustre: 25232:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 773.210134] Lustre: 25232:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.214954] Lustre: 25232:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 773.219353] Lustre: 25232:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.222217] Lustre: 25232:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 773.224517] Lustre: 25232:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.230720] Lustre: 25232:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 773.236805] Lustre: 25232:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.239353] Lustre: 25232:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 773.241635] Lustre: 25232:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.998881] Lustre: 25233:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 778.009155] Lustre: 25233:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 778.013426] Lustre: 25233:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 778.016294] Lustre: 25233:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.020328] Lustre: 25233:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 778.026522] Lustre: 25233:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.031056] Lustre: 25233:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 778.036683] Lustre: 25233:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.043847] Lustre: 25233:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 778.050707] Lustre: 25233:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.056246] Lustre: 25233:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 778.060851] Lustre: 25233:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.646165] Lustre: 26760:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 789.661188] Lustre: 26760:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 789.670658] Lustre: 26760:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.675610] Lustre: 26760:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.692407] Lustre: 26760:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.706883] Lustre: 26760:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.723465] Lustre: 26760:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 789.730343] Lustre: 26760:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.741956] Lustre: 26760:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 789.757985] Lustre: 26760:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.765896] Lustre: 26760:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.774934] Lustre: 26760:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 840.673564] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.677818] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 840.691993] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 843.404604] Lustre: server umount lustre-OST07c6 complete [ 857.081275] Lustre: server umount lustre-MDT0000 complete [ 875.434329] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 878.777527] Key type lgssc unregistered [ 879.090113] LNet: 28800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.103109] LNetError: 28800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.127812] LNet: Removed LNI 192.168.201.135@tcp [ 880.198691] Key type .llcrypt unregistered [ 880.201715] Key type ._llcrypt unregistered [ 900.601440] Key type ._llcrypt registered [ 900.607775] Key type .llcrypt registered [ 900.745951] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 902.070894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.098981] alg: No test for adler32 (adler32-zlib) [ 903.216271] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 903.494860] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 905.191249] Key type lgssc registered [ 906.537970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.159526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 914.175622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.791497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.468131] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 925.154282] LustreError: 30088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.182247] LustreError: 30088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 928.622124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.912734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 933.783540] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 941.760853] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 947.790275] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 955.199322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 955.707556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.913335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 963.325455] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 972.392366] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 988.644526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.651569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.670995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.694848] Lustre: Skipped 1 previous similar message [ 988.712747] Lustre: Skipped 1 previous similar message [ 993.763561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.787466] Lustre: Skipped 1 previous similar message [ 994.462059] Lustre: server umount lustre-OST0000 complete [ 1000.420761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1000.427601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.447390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.009829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.010865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.727955] Lustre: server umount lustre-MDT0000 complete [ 1008.044184] LustreError: 30087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.079332] LustreError: 30087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1009.985164] LustreError: 30069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735365 with bad export cookie 16914071382698367290 [ 1010.008789] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.643533] Lustre: server umount lustre-MDT0001 complete [ 1021.107262] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1023.545850] Key type lgssc unregistered [ 1023.748154] LNet: 33821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.753956] LNetError: 33821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.775126] LNet: Removed LNI 192.168.201.135@tcp [ 1024.746720] Key type .llcrypt unregistered [ 1024.749164] Key type ._llcrypt unregistered [ 1052.225722] Key type ._llcrypt registered [ 1052.231622] Key type .llcrypt registered [ 1052.339635] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1054.393588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.628137] alg: No test for adler32 (adler32-zlib) [ 1055.784981] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1056.250692] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1057.999200] Key type lgssc registered [ 1059.264265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.535897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1068.568928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.993459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.812526] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1079.267671] LustreError: 35107:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.290760] LustreError: 35107:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1083.426407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.941437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.232485] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1096.413051] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1102.435823] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.185844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.560079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.687132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1116.453307] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1125.209200] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1139.178688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1139.184015] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.197159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.211479] Lustre: Skipped 1 previous similar message [ 1139.242378] Lustre: Skipped 1 previous similar message [ 1144.300943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.314157] Lustre: Skipped 1 previous similar message [ 1145.352282] Lustre: server umount lustre-OST0000 complete [ 1149.409687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.420894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.426267] Lustre: Skipped 1 previous similar message [ 1154.531726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.908125] Lustre: server umount lustre-MDT0000 complete [ 1159.650773] LustreError: 35132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735515 with bad export cookie 10317582584196102191 [ 1159.656745] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.667180] LustreError: 35132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1159.964112] Lustre: server umount lustre-MDT0001 complete [ 1170.976551] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1174.250479] Key type lgssc unregistered [ 1174.504760] LNet: 38661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.515457] LNetError: 38661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.533143] LNet: Removed LNI 192.168.201.135@tcp [ 1175.442154] Key type .llcrypt unregistered [ 1175.444853] Key type ._llcrypt unregistered [ 1204.777505] Key type ._llcrypt registered [ 1204.778765] Key type .llcrypt registered [ 1204.823822] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1206.328991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.758957] alg: No test for adler32 (adler32-zlib) [ 1208.005538] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1208.430234] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1210.200859] Key type lgssc registered [ 1211.723355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.821235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1219.870982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.314220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.372232] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1231.863240] LustreError: 40095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.878505] LustreError: 40095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1234.238183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.531341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1238.215953] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1245.796777] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1251.997432] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1259.930109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1260.298063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.510756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1267.808668] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1275.507047] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1285.601700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1285.611126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.631769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1288.465229] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.35@tcp (stopping) [ 1290.210647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.212597] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.220761] Lustre: Skipped 1 previous similar message [ 1290.230657] Lustre: Skipped 1 previous similar message [ 1291.936432] Lustre: server umount lustre-MDT0000 complete [ 1293.575996] LustreError: 40090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.293159] LustreError: 40075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735651 with bad export cookie 15453739036341358551 [ 1295.312986] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1295.333377] LustreError: 40091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.344240] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.386638] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1295.389253] Lustre: Skipped 1 previous similar message [ 1298.687752] LustreError: 40091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.448844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1300.454183] Lustre: Skipped 1 previous similar message [ 1301.904688] Lustre: server umount lustre-MDT0001 complete [ 1317.296668] Lustre: server umount lustre-OST0000 complete [ 1335.788467] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1338.310919] Key type lgssc unregistered [ 1338.699732] LNet: 43953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.705703] LNetError: 43953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.717832] LNet: Removed LNI 192.168.201.135@tcp [ 1339.841557] Key type .llcrypt unregistered [ 1339.848222] Key type ._llcrypt unregistered [ 1364.940319] Key type ._llcrypt registered [ 1364.950098] Key type .llcrypt registered [ 1365.099200] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1366.875659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1367.028642] alg: No test for adler32 (adler32-zlib) [ 1368.190779] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1368.396389] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1370.015206] Key type lgssc registered [ 1371.673908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1379.127380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1379.138461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.480866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.819369] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1391.073832] LustreError: 45388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.100256] LustreError: 45388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1396.194620] LustreError: 45389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1397.298496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.853988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.082995] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1411.413962] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1416.717319] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1423.297256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1423.602973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.665763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1430.524969] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1438.881033] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1451.021397] Lustre: server umount lustre-OST0000 complete [ 1467.359208] Lustre: 44691:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790735807/real 1790735807] req@000000002df8ed9a x1877722502557440/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790735823 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1467.401777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.409290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1469.417772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.448463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1469.455247] Lustre: 44688:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790735809/real 1790735809] req@00000000831badf5 x1877722502558208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790735825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1470.495594] Lustre: 44691:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790735809/real 1790735809] req@000000006a77f812 x1877722502558464/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790735825 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1474.272131] Lustre: 44690:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790735814/real 1790735814] req@00000000bdd87122 x1877722502559488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790735830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1474.533167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.543270] Lustre: Skipped 2 previous similar messages [ 1475.938264] Lustre: server umount lustre-MDT0000 complete [ 1479.647453] Lustre: 44689:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790735819/real 1790735819] req@000000001d782577 x1877722502560640/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790735835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1479.657635] LustreError: 45383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.696799] LustreError: 45383:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1480.151419] LustreError: 45370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790735836 with bad export cookie 16232866477932143256 [ 1480.157379] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.160969] LustreError: 45370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1486.686793] Lustre: server umount lustre-MDT0001 complete [ 1498.108417] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1501.491494] Key type lgssc unregistered [ 1501.818975] LNet: 49096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.823573] LNetError: 49096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.840315] LNet: Removed LNI 192.168.201.135@tcp [ 1502.930543] Key type .llcrypt unregistered [ 1502.932641] Key type ._llcrypt unregistered [ 1531.745831] Key type ._llcrypt registered [ 1531.747892] Key type .llcrypt registered [ 1531.907183] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1533.504513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1533.695831] alg: No test for adler32 (adler32-zlib) [ 1534.992464] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1535.369137] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1537.063658] Key type lgssc registered [ 1538.376525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1547.292483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1547.304680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.024854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.324684] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1558.502423] LustreError: 50530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.527951] LustreError: 50530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1563.623899] LustreError: 50529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.519679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.086383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1570.249557] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1578.900961] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1584.418183] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1591.236542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1591.600577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.661947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1598.614648] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1605.909685] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1616.017193] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1620.182264] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1632.470803] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-2922h39m46s [ 1634.332639] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1636.833049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1636.843757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.855136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.932325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.943257] Lustre: Skipped 4 previous similar messages [ 1639.718893] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.35@tcp (stopping) [ 1643.374797] Lustre: server umount lustre-MDT0000 complete [ 1644.083623] LustreError: 50525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.096956] LustreError: 50525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1646.854371] LustreError: 50525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.332576] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1652.997104] LustreError: 52972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1653.020113] LustreError: 52972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1657.751270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.916282] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1658.252811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.367678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1659.316967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1665.069803] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1669.414899] Lustre: DEBUG MARKER: Wait for df (36876) ... [ 1671.435934] Lustre: DEBUG MARKER: done [ 1678.839685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1678.864141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.900863] Lustre: Skipped 2 previous similar messages [ 1678.915117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1678.941631] Lustre: Skipped 1 previous similar message [ 1683.383258] Lustre: server umount lustre-OST0000 complete [ 1690.081403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1690.087289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.103649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.107469] Lustre: Skipped 2 previous similar messages [ 1690.114472] Lustre: Skipped 2 previous similar messages [ 1695.639132] Lustre: server umount lustre-MDT0000 complete [ 1697.251817] LustreError: 50524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.286501] LustreError: 50524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1700.402508] LustreError: 50509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736056 with bad export cookie 3931052998834469838 [ 1700.417253] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1700.417955] LustreError: 50509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1700.913556] Lustre: server umount lustre-MDT0001 complete [ 1715.132641] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1718.747460] Key type lgssc unregistered [ 1719.138957] LNet: 55455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.147721] LNetError: 55455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.165155] LNet: Removed LNI 192.168.201.135@tcp [ 1720.249128] Key type .llcrypt unregistered [ 1720.252168] Key type ._llcrypt unregistered [ 1765.367215] Key type ._llcrypt registered [ 1765.369452] Key type .llcrypt registered [ 1765.468505] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 1766.585366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1766.780473] alg: No test for adler32 (adler32-zlib) [ 1767.938667] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 1768.240294] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 1769.895421] Key type lgssc registered [ 1771.187427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.495833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1779.517643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.834877] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1780.873171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1781.119770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.610084] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1796.327238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.547362] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1796.618909] Lustre: 57678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1797.071472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1802.609872] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1812.239744] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1817.708350] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1825.381404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1825.685740] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1826.010314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.641116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1833.610737] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 1840.839254] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1848.665105] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-2922h43m22s [ 1850.980489] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1860.553648] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1864.696924] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1864.809397] LustreError: 57686:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1884.973211] Lustre: lustre-MDT0000: Client 16ba4af5-f556-4212-abd7-bdec55f297b7 (at 192.168.201.35@tcp) reconnecting [ 1904.863318] LustreError: 57686:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1906.962792] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1909.228993] Lustre: DEBUG MARKER: done [ 1916.544516] Lustre: server umount lustre-OST0000 complete [ 1925.092322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.101718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.105265] Lustre: Skipped 1 previous similar message [ 1925.119619] Lustre: Skipped 1 previous similar message [ 1926.179589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.350154] Lustre: server umount lustre-MDT0000 complete [ 1933.885783] LustreError: 58422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736289 with bad export cookie 9191885714718673951 [ 1933.896618] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.901885] LustreError: 58422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1943.677591] Lustre: server umount lustre-MDT0001 complete [ 1956.665791] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 1959.435871] Key type lgssc unregistered [ 1959.705817] LNet: 62011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.711184] LNetError: 62011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.742326] LNet: Removed LNI 192.168.201.135@tcp [ 1961.751640] Key type .llcrypt unregistered [ 1961.755243] Key type ._llcrypt unregistered [ 2009.265378] Key type ._llcrypt registered [ 2009.267592] Key type .llcrypt registered [ 2009.426344] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2010.901654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2010.921220] alg: No test for adler32 (adler32-zlib) [ 2012.096865] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2012.339804] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 2014.023270] Key type lgssc registered [ 2015.739269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2020.109771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2020.130612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.436838] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2021.477595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2021.626364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.633179] Lustre: Failing over lustre-MDT0000 [ 2023.147428] Lustre: server umount lustre-MDT0000 complete [ 2029.972177] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:46:24 (1790736384) [ 2036.785498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.423637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.840915] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2053.007175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.140841] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2053.232064] Lustre: 64828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2053.560829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2057.963687] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2066.398871] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2074.135187] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.373974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2081.629987] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2081.934779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.014154] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2091.726026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2096.929891] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2104.508132] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2112.483198] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.487228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.499113] Lustre: Skipped 2 previous similar messages [ 2112.507143] Lustre: Skipped 1 previous similar message [ 2117.604345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.614384] Lustre: Skipped 3 previous similar messages [ 2118.544635] Lustre: server umount lustre-MDT0000 complete [ 2122.435911] LustreError: 66697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736478 with bad export cookie 16093187059731496659 [ 2122.446099] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.449486] LustreError: 66697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2122.748352] Lustre: server umount lustre-MDT0001 complete [ 2126.067179] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2131.122668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2139.104455] Lustre: 63741:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790736478/real 1790736478] req@00000000ab8a2c77 x1877723177852928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790736494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2139.128373] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.528316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.265233] Lustre: 63741:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790736483/real 1790736483] req@00000000bba41fc5 x1877723177853056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790736499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2147.869660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.964382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2148.895416] Lustre: 63742:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790736488/real 1790736488] req@00000000007b6bfc x1877723177853312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790736504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2151.339718] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2154.017170] Lustre: 63743:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790736493/real 1790736493] req@00000000945275b1 x1877723177853440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790736509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.978874] LustreError: 68600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.019260] LustreError: 68600:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2160.113080] LustreError: 68603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.163427] LustreError: 68603:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2163.402318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.173261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.227091] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2171.489379] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2181.224303] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2187.869530] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2194.933942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2194.944913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.957845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2195.389598] Lustre: server umount lustre-MDT0000 complete [ 2197.984902] LustreError: 66682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.988483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.023841] Lustre: Skipped 1 previous similar message [ 2199.113485] LustreError: 64821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736554 with bad export cookie 16093187059731498934 [ 2199.122156] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.122916] LustreError: 64821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2199.451213] Lustre: server umount lustre-MDT0001 complete [ 2207.105220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2216.947118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.463529] Lustre: 63743:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790736558/real 1790736558] req@0000000027d4f63f x1877723177881600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790736574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2218.517340] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.952683] LustreError: 69336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.976330] LustreError: 69336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2225.049529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.101418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2230.022774] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2230.254826] Lustre: 63742:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790736570/real 1790736570] req@000000009bba6aae x1877723177882240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790736586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2230.275706] Lustre: 63742:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2233.332507] LustreError: 71282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.358761] LustreError: 71282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2237.414086] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2237.435566] Lustre: Skipped 1 previous similar message [ 2239.931451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.515238] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2245.614621] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2254.086569] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2260.634823] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2271.210289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.221794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.233396] Lustre: Skipped 2 previous similar messages [ 2271.251527] Lustre: Skipped 2 previous similar messages [ 2274.816205] Lustre: server umount lustre-MDT0000 complete [ 2276.334944] LustreError: 71282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.367643] LustreError: 71282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2279.310784] LustreError: 66697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736635 with bad export cookie 16093187059731500313 [ 2279.328525] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.791377] Lustre: server umount lustre-MDT0001 complete [ 2287.839772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2291.722094] Lustre: DEBUG MARKER: files compared the same [ 2303.467350] Lustre: server umount lustre-OST0000 complete [ 2322.300425] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 2325.445495] Key type lgssc unregistered [ 2325.816957] LNet: 74558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.824961] LNetError: 74558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.853134] LNet: Removed LNI 192.168.201.135@tcp [ 2326.834752] Key type .llcrypt unregistered [ 2326.837764] Key type ._llcrypt unregistered [ 2354.244086] Key type ._llcrypt registered [ 2354.249225] Key type .llcrypt registered [ 2354.488970] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2356.072636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2356.233297] alg: No test for adler32 (adler32-zlib) [ 2357.390420] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2357.744540] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 2359.447276] Key type lgssc registered [ 2360.782781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2369.407487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2369.423454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.031529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.858745] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug +malloc all [ 2380.262688] LustreError: 75992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.290037] LustreError: 75992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2385.378048] LustreError: 75991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.481734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.894771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2392.280156] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug +malloc all [ 2400.123808] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2405.917326] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2412.180235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2412.469297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.539300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2418.866976] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug +malloc all [ 2428.142420] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2441.183765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2441.185871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.201186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.220217] Lustre: Skipped 1 previous similar message [ 2443.700750] Lustre: server umount lustre-OST0000 complete [ 2448.352319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2448.356772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.366964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2448.371361] Lustre: Skipped 1 previous similar message [ 2451.423985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.429075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.452028] Lustre: Skipped 1 previous similar message [ 2454.054853] Lustre: server umount lustre-MDT0000 complete [ 2456.544980] LustreError: 75986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.596825] LustreError: 75986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2459.107387] LustreError: 75972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736814 with bad export cookie 14229472623840146423 [ 2459.119411] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.725744] Lustre: server umount lustre-MDT0001 complete [ 2473.903778] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 2476.808427] Key type lgssc unregistered [ 2477.052837] LNet: 79527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.060027] LNetError: 79527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2477.093591] LNet: Removed LNI 192.168.201.135@tcp [ 2477.890164] Key type .llcrypt unregistered [ 2477.892047] Key type ._llcrypt unregistered [ 2500.903331] Key type ._llcrypt registered [ 2500.911302] Key type .llcrypt registered [ 2501.108811] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2502.195523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.339877] alg: No test for adler32 (adler32-zlib) [ 2503.446507] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2503.661048] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 2505.440280] Key type lgssc registered [ 2506.688460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2514.349128] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2514.383901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2529.951421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2535.073118] LustreError: 80933:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.135@tcp: failed processing log, type 4: rc = -110 [ 2561.759411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.264623] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2575.458733] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2583.089474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.181667] Lustre: *** cfs_fail_loc=706, val=0*** [ 2583.588173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.665235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2588.642925] LustreError: 80958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.653094] LustreError: 80958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2588.803355] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2593.761721] LustreError: 80959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.843329] LustreError: 80958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.863079] LustreError: 80958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2598.426838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.958626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2604.181521] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2611.124249] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2616.550896] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2619.362265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2619.368753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.376172] LustreError: Skipped 1 previous similar message [ 2619.378053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.410748] Lustre: Skipped 1 previous similar message [ 2624.492510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2624.498839] Lustre: Skipped 1 previous similar message [ 2624.952291] Lustre: server umount lustre-OST0000 complete [ 2629.601937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.604959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.610328] Lustre: Skipped 1 previous similar message [ 2629.620401] Lustre: Skipped 2 previous similar messages [ 2634.724031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.730014] Lustre: Skipped 1 previous similar message [ 2635.176740] Lustre: server umount lustre-MDT0000 complete [ 2639.154812] LustreError: 80941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790736995 with bad export cookie 8899314950656383112 [ 2639.163072] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.168487] LustreError: 80941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2640.035309] Lustre: server umount lustre-MDT0001 complete [ 2653.162596] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 2656.497695] Key type lgssc unregistered [ 2656.821454] LNet: 84546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.837257] LNetError: 84546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2657.900206] LNet: Removed LNI 192.168.201.135@tcp [ 2659.389209] Key type .llcrypt unregistered [ 2659.391178] Key type ._llcrypt unregistered [ 2687.473583] Key type ._llcrypt registered [ 2687.476506] Key type .llcrypt registered [ 2687.611956] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2688.739140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.756983] alg: No test for adler32 (adler32-zlib) [ 2690.055494] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 2690.361237] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 2692.079380] Key type lgssc registered [ 2693.684485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2701.516888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2701.534224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.013914] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2714.706097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.440376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.736821] LustreError: 86669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.781540] LustreError: 86669:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2722.174451] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2725.859375] LustreError: 86668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.898620] LustreError: 86668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2731.012696] LustreError: 86673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.043040] LustreError: 86673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2734.050700] LustreError: 86672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.085065] LustreError: 86672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2734.972318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2735.621347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2740.720461] LustreError: 86673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.743472] LustreError: 86673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2742.432412] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2750.055286] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.071187] Lustre: MGS: Client 14e87449-4f10-480e-b196-2dcd3aaf0aad (at 0@lo) reconnecting [ 2750.093253] LustreError: 86683:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2750.327777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.404327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2754.647857] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2770.923736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.932719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2770.938891] Lustre: Skipped 1 previous similar message [ 2775.102500] Lustre: server umount lustre-OST0000 complete [ 2781.152841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2781.157284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.162619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2781.176845] Lustre: Skipped 1 previous similar message [ 2781.179572] Lustre: Skipped 1 previous similar message [ 2785.314584] Lustre: server umount lustre-MDT0000 complete [ 2786.283280] LustreError: 88527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.293744] LustreError: 88527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2789.226229] Lustre: server umount lustre-MDT0001 complete [ 2792.359841] Lustre: server umount MGS complete [ 2803.896814] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:59:18 (1790737158) [ 2835.840854] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 2847.348421] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 2884.664530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.626269] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2905.569753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.464655] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2906.557608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2906.712212] Lustre: lustre-MDT0001: new disk, initializing [ 2906.840129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2906.898446] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2906.916441] Lustre: 93873:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2906.926743] LustreError: 93873:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2914.214974] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2926.671929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2926.902552] Lustre: lustre-OST0000: new disk, initializing [ 2926.905925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2926.912766] Lustre: 94641:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2927.041078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.746391] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2935.907221] Lustre: 95000:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000df9ef1d6 x1877723888788352/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790737307 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2945.825398] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2952.486449] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2952.497763] Lustre: MGS: Client b1891b24-f5c0-44da-8e41-fb84e4552fcd (at 0@lo) reconnecting [ 2952.562374] Lustre: 95686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2952.580269] LustreError: 93859:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2952.588376] LustreError: 93859:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2952.609045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2952.703228] Lustre: lustre-MDT0000: new disk, initializing [ 2952.840263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.871966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2952.902353] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2953.952136] Lustre: 95710:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006862f531 x1877723888795136/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790737325 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2956.740659] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 2959.073885] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2963.050481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2963.056408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2963.144544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2989.025838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.027531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.039358] LustreError: Skipped 1 previous similar message [ 2989.055653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2989.064340] Lustre: Skipped 2 previous similar messages [ 2992.804337] Lustre: server umount lustre-OST0000 complete [ 3000.801126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.804476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3000.840702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.845948] Lustre: Skipped 2 previous similar messages [ 3000.849224] Lustre: Skipped 2 previous similar messages [ 3002.486705] Lustre: server umount lustre-MDT0000 complete [ 3012.017523] Lustre: server umount lustre-MDT0001 complete [ 3015.371125] Lustre: server umount MGS complete [ 3025.413578] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:03:00 (1790737380) [ 3042.795453] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3045.615052] Key type lgssc unregistered [ 3045.943294] LNet: 98209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3045.958628] LNetError: 98209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3045.977864] LNet: Removed LNI 192.168.201.135@tcp [ 3046.882223] Key type .llcrypt unregistered [ 3046.885913] Key type ._llcrypt unregistered [ 3064.932611] Key type ._llcrypt registered [ 3064.934832] Key type .llcrypt registered [ 3065.062694] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3066.539482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.578880] alg: No test for adler32 (adler32-zlib) [ 3067.708955] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3068.008951] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3069.727289] Key type lgssc registered [ 3071.150050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3078.789054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3078.813780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.226387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.666684] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3090.404987] LustreError: 99465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.429316] LustreError: 99465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3094.997218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.468220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.261690] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3107.963521] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3114.852294] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3122.425555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.990856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.059734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3129.525030] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3136.122420] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3146.725039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3146.730369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.734589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.818879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3150.826046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.829448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3152.881740] Lustre: server umount lustre-OST0000 complete [ 3161.059791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.090566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.096220] Lustre: Skipped 2 previous similar messages [ 3163.619943] Lustre: server umount lustre-MDT0000 complete [ 3166.178694] LustreError: 99460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.203924] LustreError: 99460:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3167.609078] LustreError: 99445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737523 with bad export cookie 10091974239330801561 [ 3167.618548] LustreError: 99445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3167.623241] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3168.049744] Lustre: server umount lustre-MDT0001 complete [ 3179.239782] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3182.106494] Key type lgssc unregistered [ 3182.473719] LNet: 102997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3182.490823] LNetError: 102997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3182.510041] LNet: Removed LNI 192.168.201.135@tcp [ 3183.305084] Key type .llcrypt unregistered [ 3183.309297] Key type ._llcrypt unregistered [ 3199.729910] Key type ._llcrypt registered [ 3199.731303] Key type .llcrypt registered [ 3199.817652] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3200.912567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3201.047329] alg: No test for adler32 (adler32-zlib) [ 3202.232695] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3202.524378] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3204.199742] Key type lgssc registered [ 3205.822677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3212.509624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3212.516561] LustreError: 104101:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3212.557626] LustreError: 104111:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3212.562567] LustreError: 104101:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4960 [ 3212.570492] LustreError: 104101:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3212.574564] LustreError: 104101:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3212.578345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.255782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.150359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.400736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3226.609487] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3231.473367] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing lsmod [ 3233.982549] LustreError: 105313:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3234.010200] LustreError: 105317:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3234.016025] LustreError: 105313:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4974 [ 3234.025885] LustreError: 105313:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3234.037340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3234.044149] LustreError: 105313:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3234.441942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.856415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3237.868516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.886847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3239.905267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.906673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3239.924098] Lustre: Skipped 1 previous similar message [ 3239.938750] Lustre: Skipped 2 previous similar messages [ 3243.601265] Lustre: server umount lustre-MDT0000 complete [ 3245.028284] LustreError: 104145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.056357] LustreError: 104145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3247.294873] LustreError: 104125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737603 with bad export cookie 118484681115691373 [ 3247.303081] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.307820] LustreError: 104125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3247.695957] Lustre: server umount lustre-MDT0001 complete [ 3262.416714] Lustre: server umount lustre-OST0000 complete [ 3270.716556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.103335] LustreError: 106381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.124628] LustreError: 106381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3271.207615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.127917] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3276.261932] LustreError: 106382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.356483] LustreError: 106381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.075272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.414273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3290.586816] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3298.735426] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3304.588187] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3311.292154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3311.757801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.826776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3318.142374] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3326.469045] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3340.769850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3340.785070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.828634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.829645] Lustre: Skipped 1 previous similar message [ 3340.848093] Lustre: Skipped 1 previous similar message [ 3342.269756] Lustre: server umount lustre-OST0000 complete [ 3351.023050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.025499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3351.042063] Lustre: Skipped 1 previous similar message [ 3351.061877] Lustre: Skipped 1 previous similar message [ 3353.694775] Lustre: server umount lustre-MDT0000 complete [ 3356.142385] LustreError: 106376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.160142] LustreError: 106376:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3357.884164] LustreError: 106361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737713 with bad export cookie 118484681115692066 [ 3357.899616] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.899776] LustreError: 106361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3358.165825] Lustre: server umount lustre-MDT0001 complete [ 3369.022830] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3371.921446] Key type lgssc unregistered [ 3372.174706] LNet: 109938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.187133] LNetError: 109938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3372.218850] LNet: Removed LNI 192.168.201.135@tcp [ 3373.047523] Key type .llcrypt unregistered [ 3373.049675] Key type ._llcrypt unregistered [ 3395.994353] Key type ._llcrypt registered [ 3395.995933] Key type .llcrypt registered [ 3396.071964] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3396.971693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3397.087965] alg: No test for adler32 (adler32-zlib) [ 3398.162443] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3398.411639] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3400.072137] Key type lgssc registered [ 3401.188911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3408.719323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3408.732751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.225849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.556209] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3420.651559] LustreError: 111370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.666522] LustreError: 111370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3425.012807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.522916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3430.493720] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3438.292343] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3443.504807] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3450.257660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.698057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.821579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3457.761049] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3466.957984] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3476.000041] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3479.932696] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3486.692476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.694882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.697843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.710458] Lustre: Skipped 1 previous similar message [ 3490.980730] Lustre: server umount lustre-OST0000 complete [ 3496.929808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.932500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3496.937066] Lustre: Skipped 1 previous similar message [ 3496.944424] Lustre: Skipped 2 previous similar messages [ 3501.403052] Lustre: server umount lustre-MDT0000 complete [ 3502.053933] LustreError: 112611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.054693] LustreError: 111366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.251978] LustreError: 112104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737860 with bad export cookie 8570104869823512873 [ 3504.262491] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3504.533321] Lustre: server umount lustre-MDT0001 complete [ 3515.175787] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3518.026183] Key type lgssc unregistered [ 3518.334477] LNet: 115007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3518.348556] LNetError: 115007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3518.370063] LNet: Removed LNI 192.168.201.135@tcp [ 3519.364574] Key type .llcrypt unregistered [ 3519.366618] Key type ._llcrypt unregistered [ 3538.852598] Key type ._llcrypt registered [ 3538.854608] Key type .llcrypt registered [ 3539.001924] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3540.194915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3540.302717] alg: No test for adler32 (adler32-zlib) [ 3541.379868] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3541.611705] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3543.335990] Key type lgssc registered [ 3544.779919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3554.111506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3554.134447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.702323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.956537] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3566.054688] LustreError: 116242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.065466] LustreError: 116242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3568.979962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.398944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3573.621356] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3580.882982] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3586.830460] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3594.112714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3594.442423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.553264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3601.061506] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3608.189216] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3620.333158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.335991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3620.347646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3620.347954] Lustre: Skipped 1 previous similar message [ 3624.290524] Lustre: server umount lustre-OST0000 complete [ 3630.561916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.570216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3630.582603] Lustre: Skipped 1 previous similar message [ 3630.595116] Lustre: Skipped 2 previous similar messages [ 3634.991345] Lustre: server umount lustre-MDT0000 complete [ 3635.681018] LustreError: 117285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.698538] LustreError: 117285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3638.830259] LustreError: 116224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790737994 with bad export cookie 11463671670689948681 [ 3638.830975] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3638.836387] LustreError: 116224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3639.302168] Lustre: server umount lustre-MDT0001 complete [ 3649.219469] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3651.644495] Key type lgssc unregistered [ 3651.977227] LNet: 119777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3651.988909] LNetError: 119777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3652.011143] LNet: Removed LNI 192.168.201.135@tcp [ 3652.831565] Key type .llcrypt unregistered [ 3652.833481] Key type ._llcrypt unregistered [ 3681.294914] Key type ._llcrypt registered [ 3681.297287] Key type .llcrypt registered [ 3681.438251] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3682.841392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3682.912088] alg: No test for adler32 (adler32-zlib) [ 3683.977440] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3684.209755] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3685.912553] Key type lgssc registered [ 3687.057205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3693.972445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3694.003220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.464701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.509030] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3705.840611] LustreError: 121214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.874563] LustreError: 121214:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3707.079181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.530555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3711.367241] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3720.086973] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3726.449393] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3732.676288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.055968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.930894] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3740.230406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3747.134543] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3757.737167] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3761.683421] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3761.695127] Lustre: Skipped 2 previous similar messages [ 3763.829976] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3763.871705] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3766.187341] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3766.190699] Lustre: Skipped 2 previous similar messages [ 3773.018765] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3773.023965] Lustre: Skipped 1 previous similar message [ 3778.597427] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.35@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3778.602707] Lustre: Skipped 2 previous similar messages [ 3783.218548] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.35@tcp 192.168.0.[2,10]@tcp [ 3787.630559] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.35/24@tcp in log lustre-MDT0000 [ 3787.635503] Lustre: Skipped 2 previous similar messages [ 3790.916349] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.35/24@tcp [ 3790.925586] Lustre: Skipped 1 previous similar message [ 3803.619470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.622492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.623257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.644520] Lustre: Skipped 1 previous similar message [ 3805.468640] Lustre: server umount lustre-OST0000 complete [ 3809.761093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3809.769919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.790851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3809.798726] Lustre: Skipped 1 previous similar message [ 3813.859687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.861715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.644666] Lustre: server umount lustre-MDT0000 complete [ 3818.981931] LustreError: 121215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3819.000146] LustreError: 121215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3819.220658] LustreError: 121195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738175 with bad export cookie 13432854054709549671 [ 3819.228659] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3819.244696] LustreError: 121195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3819.736771] Lustre: server umount lustre-MDT0001 complete [ 3829.556381] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3832.499365] Key type lgssc unregistered [ 3832.940518] LNet: 126200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3832.946444] LNetError: 126200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3832.974556] LNet: Removed LNI 192.168.201.135@tcp [ 3834.118788] Key type .llcrypt unregistered [ 3834.122737] Key type ._llcrypt unregistered [ 3864.559914] Key type ._llcrypt registered [ 3864.562073] Key type .llcrypt registered [ 3864.713963] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 3866.723366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3866.768169] alg: No test for adler32 (adler32-zlib) [ 3867.857627] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 3868.237279] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 3870.007313] Key type lgssc registered [ 3871.768671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3880.496864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3880.534363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.171944] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3882.207490] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.35/24@tcp [ 3882.229429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3887.707820] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3891.689871] LustreError: 127778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.721950] LustreError: 127778:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3896.078761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.293411] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3896.321430] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.35/24@tcp [ 3896.373070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3900.663638] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3907.961410] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3912.783470] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3918.902942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.145326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.287577] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 3926.337710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3931.855821] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3941.858000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.860628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3941.878153] Lustre: Skipped 1 previous similar message [ 3941.897451] Lustre: Skipped 1 previous similar message [ 3946.979329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3946.989175] Lustre: Skipped 1 previous similar message [ 3947.577829] Lustre: server umount lustre-OST0000 complete [ 3952.097951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.106229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3952.111660] Lustre: Skipped 1 previous similar message [ 3957.217028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3957.225576] Lustre: Skipped 2 previous similar messages [ 3962.337078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.347193] Lustre: Skipped 1 previous similar message [ 3965.407273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3965.623181] Lustre: server umount lustre-MDT0000 complete [ 3967.462529] LustreError: 130412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3967.496953] LustreError: 130412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3968.894932] LustreError: 127758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738324 with bad export cookie 16854419300425565541 [ 3968.902856] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.904896] LustreError: 127758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3969.171336] Lustre: server umount lustre-MDT0001 complete [ 3981.178054] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 3984.071220] Key type lgssc unregistered [ 3984.531284] LNet: 131366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3984.560794] LNetError: 131366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3985.573745] LNet: Removed LNI 192.168.201.135@tcp [ 3986.815253] Key type .llcrypt unregistered [ 3986.816940] Key type ._llcrypt unregistered [ 4015.002180] Key type ._llcrypt registered [ 4015.006381] Key type .llcrypt registered [ 4015.124840] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 4028.619732] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4029.369044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4029.464157] alg: No test for adler32 (adler32-zlib) [ 4030.451582] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 4030.687386] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 4032.383245] Key type lgssc registered [ 4033.413807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4065.725037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4065.772916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.035416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4067.062765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4067.153110] Lustre: lustre-MDT0000: new disk, initializing [ 4067.290269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.316462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4071.916749] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4081.106578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.183632] Lustre: 134783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4081.212800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4081.215320] Lustre: Skipped 1 previous similar message [ 4081.252766] Lustre: lustre-MDT0001: new disk, initializing [ 4081.309302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4081.338327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4081.345535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4085.301778] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4094.862346] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4099.548161] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4107.582164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.800476] Lustre: lustre-OST0000: new disk, initializing [ 4107.803269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4107.807634] Lustre: 136749:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4107.871702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.135290] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4115.515497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4115.532798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4115.661231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4121.785807] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4125.795510] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4126.042671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4130.458750] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4130.670666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4136.434095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.437994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4136.451958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4136.460513] Lustre: Skipped 1 previous similar message [ 4138.805575] Lustre: server umount lustre-OST0000 complete [ 4143.073944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4143.079680] LustreError: Skipped 1 previous similar message [ 4143.083989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.106028] Lustre: Skipped 1 previous similar message [ 4143.112582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.657831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.658130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.682085] Lustre: Skipped 1 previous similar message [ 4151.776403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4156.383153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4156.541557] Lustre: server umount lustre-MDT0000 complete [ 4160.076756] LustreError: 134775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738515 with bad export cookie 5785538006630523242 [ 4160.082252] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4160.086806] LustreError: 134775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4160.429947] Lustre: server umount lustre-MDT0001 complete [ 4168.232476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.626143] LustreError: 138967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.653340] LustreError: 138967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4168.734259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.337710] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4173.801800] LustreError: 138968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.914115] LustreError: 138967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.922670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.243368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4185.256873] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4191.853293] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4196.875392] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4203.144372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4203.592783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.192523] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4215.898693] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4220.414735] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4220.599787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4225.873463] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4226.042592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4234.647203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.987338] Lustre: lustre-OST0001: new disk, initializing [ 4234.993520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4235.005682] Lustre: 142335:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4235.180881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4242.983542] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4244.590910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4244.603988] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4244.705732] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4251.126622] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4251.360304] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4256.314837] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4256.560180] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4262.224073] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94d3854e6800.ost_server_uuid 50 [ 4263.755856] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94d3854e6800.ost_server_uuid in IDLE state after 0 sec [ 4275.688322] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4275.700287] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.706954] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4275.717281] Lustre: Skipped 1 previous similar message [ 4275.738979] Lustre: Skipped 2 previous similar messages [ 4276.978681] Lustre: server umount lustre-OST0001 complete [ 4280.806735] LustreError: 140787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.812229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4280.812988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.823558] LustreError: 140787:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4280.869856] LustreError: Skipped 1 previous similar message [ 4285.922798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4285.929791] LustreError: 140787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.936713] Lustre: Skipped 3 previous similar messages [ 4285.955977] LustreError: 140787:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4286.552246] Lustre: server umount lustre-OST0000 complete [ 4291.044480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.067584] Lustre: Skipped 2 previous similar messages [ 4304.351410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4304.672301] Lustre: server umount lustre-MDT0000 complete [ 4306.400472] LustreError: 138967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.423549] LustreError: 138967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4308.462041] LustreError: 138949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738664 with bad export cookie 5785538006630524124 [ 4308.490031] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.490860] LustreError: 138949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4308.851291] Lustre: server umount lustre-MDT0001 complete [ 4329.290779] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 4332.585759] Key type lgssc unregistered [ 4332.935663] LNet: 145343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4332.940663] LNetError: 145343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4332.957160] LNet: Removed LNI 192.168.201.135@tcp [ 4334.000257] Key type .llcrypt unregistered [ 4334.001757] Key type ._llcrypt unregistered [ 4375.225176] Key type ._llcrypt registered [ 4375.229870] Key type .llcrypt registered [ 4375.338926] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 4389.873798] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4390.427260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4390.522056] alg: No test for adler32 (adler32-zlib) [ 4391.627268] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 4391.924549] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 4393.599206] Key type lgssc registered [ 4394.818744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4427.974212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4428.015723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.329142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4429.361165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4429.421480] Lustre: lustre-MDT0000: new disk, initializing [ 4429.549426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.579329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4434.878931] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4445.798665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.003985] Lustre: 149512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4446.105090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4446.110143] Lustre: Skipped 1 previous similar message [ 4446.178701] Lustre: lustre-MDT0001: new disk, initializing [ 4446.289345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4446.322189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4446.337269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4450.500929] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4460.616683] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4465.332708] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4472.407476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4472.669967] Lustre: lustre-OST0000: new disk, initializing [ 4472.672678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4472.681601] Lustre: 151477:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4472.739097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.657353] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4482.640170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4482.664864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4482.797557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4489.507778] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4494.186486] Lustre: Failing over lustre-OST0000 [ 4494.480331] Lustre: server umount lustre-OST0000 complete [ 4512.737574] Lustre: 147726:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790738852/real 1790738852] req@00000000aa94ab5b x1877725672407936/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1790738868 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4512.766496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.787361] Lustre: 147728:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790738854/real 1790738854] req@00000000a6dc0c7a x1877725672409216/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790738870 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4514.842528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.939898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.206848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4516.221909] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4519.269370] Lustre: 147728:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790738859/real 1790738859] req@00000000de3db0c0 x1877725672410240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790738875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4519.315457] Lustre: 147728:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4522.247845] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4522.421639] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4524.420917] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4524.421360] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4525.545201] Lustre: 147726:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790738864/real 1790738864] req@0000000005473c3f x1877725672411392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790738880 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4525.581327] Lustre: 147726:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4526.358893] Lustre: Failing over lustre-MDT0000 [ 4526.808728] Lustre: server umount lustre-MDT0000 complete [ 4527.395859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.419909] Lustre: Skipped 2 previous similar messages [ 4532.477618] LustreError: 149518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.506238] LustreError: 149518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4537.597466] LustreError: 150599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.609569] LustreError: 150599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4542.723715] LustreError: 149518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.753884] LustreError: 149518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4543.968816] Lustre: 147726:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790738883/real 1790738883] req@00000000d9f759e3 x1877725672423808/t0(0) o400->MGC192.168.201.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790738899 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4543.991133] Lustre: 147726:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4543.995478] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4548.663940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4548.870909] LustreError: 150599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4548.931401] LustreError: 150599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4552.957899] LustreError: 149517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.984220] LustreError: 149517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4553.186210] LustreError: 147725:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ba21a765 x1877725672429312/t0(0) o250->MGC192.168.201.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4553.599345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4553.660532] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4553.981918] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4558.668416] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4558.832197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4558.841533] Lustre: Skipped 1 previous similar message [ 4558.865198] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4558.911651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4569.060424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4569.071225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4569.079027] Lustre: Skipped 1 previous similar message [ 4569.108807] Lustre: Skipped 1 previous similar message [ 4571.994650] Lustre: server umount lustre-OST0000 complete [ 4579.296704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4579.307875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.318786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4579.337686] Lustre: Skipped 1 previous similar message [ 4584.419054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4584.426137] Lustre: Skipped 2 previous similar messages [ 4589.548399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4589.565055] Lustre: Skipped 1 previous similar message [ 4591.072308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4591.393338] Lustre: server umount lustre-MDT0000 complete [ 4594.667406] LustreError: 152666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.687144] LustreError: 152666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4594.810549] LustreError: 153278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790738950 with bad export cookie 16993756683912102819 [ 4594.819600] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4594.824396] LustreError: 153278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4595.102604] Lustre: server umount lustre-MDT0001 complete [ 4607.253401] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 4610.462181] Key type lgssc unregistered [ 4610.887665] LNet: 155135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4610.894831] LNetError: 155135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4610.921189] LNet: Removed LNI 192.168.201.135@tcp [ 4611.883560] Key type .llcrypt unregistered [ 4611.886269] Key type ._llcrypt unregistered [ 4648.575257] Key type ._llcrypt registered [ 4648.577405] Key type .llcrypt registered [ 4648.625915] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4649.265804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4649.505914] alg: No test for adler32 (adler32-zlib) [ 4650.479319] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 4650.927851] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 4652.665904] Key type lgssc registered [ 4655.009191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4663.755414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4663.777685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.422487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.965925] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4675.561467] LustreError: 156675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.588552] LustreError: 156675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4680.676413] LustreError: 156674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.052194] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4685.027519] LustreError: 156704:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4685.041698] LustreError: 156704:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4685.051080] LustreError: 156704:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4685.408469] Lustre: server umount lustre-MDT0000 complete [ 4698.284275] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 4709.480321] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 4747.041983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.252758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4747.291408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4747.352337] Lustre: lustre-MDT0000: new disk, initializing [ 4747.399738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4747.416075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4753.313651] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4765.042462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.362730] Lustre: 160008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4765.481332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4765.485254] Lustre: Skipped 1 previous similar message [ 4765.569206] Lustre: lustre-MDT0001: new disk, initializing [ 4765.714666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4765.753462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4765.763452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4769.703261] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4778.224424] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4783.184664] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4789.589710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4789.768279] Lustre: lustre-OST0000: new disk, initializing [ 4789.771481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4789.775144] Lustre: 161976:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4789.864828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4796.046882] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 4797.039268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4797.051041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4797.162891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4805.658295] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4809.414090] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4811.169805] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:32:45 (1790739165) [ 5026.784077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5026.799301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.821595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5027.810584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5027.811939] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.949460] LustreError: 160021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5032.961114] LustreError: 160021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5033.055365] Lustre: server umount lustre-MDT0000 complete [ 5036.409242] LustreError: 160000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739392 with bad export cookie 14449483799351894874 [ 5036.409912] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5036.421689] LustreError: 160000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5038.048283] LustreError: 163744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.052220] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.115989] Lustre: Skipped 1 previous similar message [ 5038.128627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5038.130986] Lustre: Skipped 2 previous similar messages [ 5041.055900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5042.764441] Lustre: server umount lustre-MDT0001 complete [ 5052.616369] Lustre: server umount lustre-OST0000 complete [ 5063.953282] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 5077.050413] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5114.331124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.516889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5114.542880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5114.625189] Lustre: lustre-MDT0000: new disk, initializing [ 5114.666077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.675409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5118.869366] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5130.991893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5131.146864] Lustre: 166635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5131.249952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5131.267663] Lustre: Skipped 1 previous similar message [ 5131.387567] Lustre: lustre-MDT0001: new disk, initializing [ 5131.492198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5131.533052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5131.542650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5137.041820] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5148.545609] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5152.931936] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5158.539683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5158.878479] Lustre: lustre-OST0000: new disk, initializing [ 5158.884855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5158.889729] Lustre: 168599:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5158.988281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5164.427683] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5166.649383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5166.661952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5166.758198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5173.465597] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5178.879112] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5179.186531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5184.219790] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5184.595866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5187.556508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5187.560577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.567453] LustreError: Skipped 1 previous similar message [ 5187.578142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5187.587106] Lustre: Skipped 1 previous similar message [ 5187.618419] Lustre: Skipped 1 previous similar message [ 5192.689209] Lustre: server umount lustre-OST0000 complete [ 5197.794295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.814584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.814761] Lustre: Skipped 1 previous similar message [ 5197.826378] Lustre: Skipped 1 previous similar message [ 5202.639907] Lustre: server umount lustre-MDT0000 complete [ 5202.923560] LustreError: 166645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.940123] LustreError: 166645:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5206.870872] LustreError: 166627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739562 with bad export cookie 14449483799352053704 [ 5206.874048] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5206.882812] LustreError: 166627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5207.216076] Lustre: server umount lustre-MDT0001 complete [ 5215.643335] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:39:30 (1790739570) [ 5232.280472] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 5242.892086] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5278.627446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.876468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5278.896871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5278.959700] Lustre: lustre-MDT0000: new disk, initializing [ 5279.011143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.024630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5283.328298] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5293.166912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.267803] Lustre: 173356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5293.524040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5297.675962] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5308.787712] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5315.458041] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5323.896904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.306941] Lustre: lustre-OST0000: new disk, initializing [ 5324.316477] Lustre: Skipped 1 previous similar message [ 5324.323785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5324.332065] Lustre: Skipped 2 previous similar messages [ 5324.342953] Lustre: 175334:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5330.596654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5330.615892] Lustre: Skipped 1 previous similar message [ 5330.620331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5330.734522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5332.008319] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5341.233846] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5351.396037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5351.400747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.403037] LustreError: Skipped 1 previous similar message [ 5351.403703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5351.419364] Lustre: Skipped 1 previous similar message [ 5351.438696] Lustre: Skipped 1 previous similar message [ 5363.167153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5363.301561] Lustre: server umount lustre-OST0000 complete [ 5374.802458] Lustre: server umount lustre-MDT0000 complete [ 5376.996195] LustreError: 173370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.021682] LustreError: 173370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5379.087124] LustreError: 173350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739734 with bad export cookie 14449483799352054586 [ 5379.096292] LustreError: 173350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5379.098527] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5379.763826] Lustre: server umount lustre-MDT0001 complete [ 5390.490909] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:42:24 (1790739744) [ 5407.982736] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 5418.148619] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5457.028795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.360807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5457.407469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5457.536157] Lustre: lustre-MDT0000: new disk, initializing [ 5457.664426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5457.676391] Lustre: Skipped 2 previous similar messages [ 5457.718496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5463.439721] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5474.951647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.129500] Lustre: 179661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5475.568714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5480.172424] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5490.844541] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5497.743827] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5505.125457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5505.521506] Lustre: 181639:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5512.407598] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5512.812422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5512.950164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5521.162859] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5533.669277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.673196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5533.695040] Lustre: Skipped 3 previous similar messages [ 5533.703145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5533.717252] Lustre: Skipped 9 previous similar messages [ 5535.570433] Lustre: server umount lustre-OST0000 complete [ 5542.372690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5548.252771] Lustre: server umount lustre-MDT0000 complete [ 5549.024929] LustreError: 179674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5549.066737] LustreError: 179674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5552.442528] LustreError: 179654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790739908 with bad export cookie 14449483799352055671 [ 5552.456966] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5552.463167] LustreError: 179654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5552.735851] Lustre: server umount lustre-MDT0001 complete [ 5566.414948] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 5569.995768] Key type lgssc unregistered [ 5570.392052] LNet: 183495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5570.404542] LNetError: 183495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5570.424615] LNet: Removed LNI 192.168.201.135@tcp [ 5571.615361] Key type .llcrypt unregistered [ 5571.616848] Key type ._llcrypt unregistered [ 5598.381646] Key type ._llcrypt registered [ 5598.384553] Key type .llcrypt registered [ 5598.524233] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5599.968389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5600.201369] alg: No test for adler32 (adler32-zlib) [ 5601.337464] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5601.630359] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 5603.375281] Key type lgssc registered [ 5604.971625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5613.742097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5613.763595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5615.348716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.994386] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5625.829477] LustreError: 184927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.850299] LustreError: 184927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5628.474369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.829612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5633.383345] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5640.523226] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5646.076810] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5653.168321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5653.666911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5660.867020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5662.534929] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5671.590468] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5690.852850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5690.856697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5690.870665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5691.881783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5691.892925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.894703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5691.921551] Lustre: Skipped 1 previous similar message [ 5697.060447] Lustre: server umount lustre-OST0000 complete [ 5702.115401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5702.135290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5707.236482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5707.238884] Lustre: Skipped 2 previous similar messages [ 5707.538793] Lustre: server umount lustre-MDT0000 complete [ 5711.196493] LustreError: 184907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740067 with bad export cookie 17560599082912537777 [ 5711.202457] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5711.209201] LustreError: 184907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5711.706198] Lustre: server umount lustre-MDT0001 complete [ 5723.138397] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 5726.236629] Key type lgssc unregistered [ 5726.517180] LNet: 188483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5726.530108] LNetError: 188483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5726.567542] LNet: Removed LNI 192.168.201.135@tcp [ 5727.445313] Key type .llcrypt unregistered [ 5727.448177] Key type ._llcrypt unregistered [ 5751.658305] Key type ._llcrypt registered [ 5751.660950] Key type .llcrypt registered [ 5751.752593] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 5753.018698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5753.101699] alg: No test for adler32 (adler32-zlib) [ 5754.247617] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 5754.556365] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 5756.295223] Key type lgssc registered [ 5758.222467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5767.297883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5767.327174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.866832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5773.363506] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5778.403132] LustreError: 189916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.432367] LustreError: 189916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5781.827369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5782.094028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5786.962422] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5795.023421] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5801.456909] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5808.468615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5808.844060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.010370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5815.278980] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5823.901838] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5831.458262] Lustre: lustre-OST0000: Not available for connect from 192.168.201.35@tcp (stopping) [ 5831.652336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5831.654158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.553328] Lustre: lustre-OST0000: Not available for connect from 192.168.201.35@tcp (stopping) [ 5836.565070] Lustre: Skipped 2 previous similar messages [ 5836.775310] LustreError: 191745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.917907] Lustre: server umount lustre-OST0000 complete [ 5845.291337] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff94d384522800.ost_server_uuid 50 [ 5846.582524] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5862.305317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.310364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5862.314676] Lustre: Skipped 2 previous similar messages [ 5862.331481] Lustre: Skipped 1 previous similar message [ 5867.501080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5867.518498] Lustre: Skipped 1 previous similar message [ 5867.809810] Lustre: server umount lustre-MDT0000 complete [ 5871.702914] LustreError: 189896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740227 with bad export cookie 4154409780982771773 [ 5871.712042] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5871.717372] LustreError: 189896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5872.115062] Lustre: server umount lustre-MDT0001 complete [ 5880.695835] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:50:35 (1790740235) [ 5886.999385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.558871] LustreError: 193839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.572849] LustreError: 193839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5887.634705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5892.595269] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5893.090566] LustreError: 193840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.190659] LustreError: 193839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.994648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.623772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5906.349433] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5914.125582] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5918.973514] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5925.568892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5925.767440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.533637] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5933.053312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5939.068734] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5947.722309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5948.089492] Lustre: lustre-OST0001: new disk, initializing [ 5948.092552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5948.102223] Lustre: 196679:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5948.209353] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5955.062383] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 5956.240283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5956.252627] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5956.354192] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5964.944499] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5968.872639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5968.879471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5968.897453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5971.951046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.290116] Lustre: server umount lustre-OST0000 complete [ 5975.822332] LustreError: 196288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.710462] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5982.004905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5986.849414] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5987.066571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5992.426327] LustreError: 195659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.449044] LustreError: 195659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6002.667363] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.672348] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6002.682745] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6002.694587] LustreError: Skipped 1 previous similar message [ 6002.700312] Lustre: Skipped 4 previous similar messages [ 6007.667444] Lustre: server umount lustre-OST0001 complete [ 6012.896384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6012.906654] Lustre: Skipped 1 previous similar message [ 6017.463721] Lustre: server umount lustre-MDT0000 complete [ 6021.592705] LustreError: 193821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740377 with bad export cookie 4154409780982772851 [ 6021.599419] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6021.606956] LustreError: 193821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6022.102397] Lustre: server umount lustre-MDT0001 complete [ 6050.001944] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:53:24 (1790740404) [ 6056.418307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.823376] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6056.905926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6057.151474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.412926] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6069.893209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.027756] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6070.087733] Lustre: 200121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6075.031144] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6083.431202] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6088.696324] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6096.022779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.294603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6102.011068] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6103.835094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6114.720864] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6120.853421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.896533] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6121.240334] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6121.249565] Lustre: Skipped 2 previous similar messages [ 6127.136694] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6134.796821] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6139.872719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6139.880426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6139.888163] LustreError: Skipped 1 previous similar message [ 6139.889971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6139.898059] Lustre: Skipped 2 previous similar messages [ 6139.915052] Lustre: Skipped 3 previous similar messages [ 6143.526351] Lustre: server umount lustre-OST0000 complete [ 6145.807854] LustreError: 202404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.824743] LustreError: 202404:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6161.377556] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6161.403102] LustreError: Skipped 1 previous similar message [ 6164.747758] Lustre: server umount lustre-OST0001 complete [ 6175.071235] Lustre: server umount lustre-MDT0000 complete [ 6178.824306] LustreError: 200114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740534 with bad export cookie 4154409780982774258 [ 6178.828622] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.843729] LustreError: 200114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6179.305662] Lustre: server umount lustre-MDT0001 complete [ 6201.556559] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:55:56 (1790740556) [ 6217.273639] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 6228.059966] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 6267.724026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.061528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6268.120403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6268.208775] Lustre: lustre-MDT0000: new disk, initializing [ 6268.319053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.369212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6273.303489] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6284.720786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6284.820286] Lustre: 208216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6284.867552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6284.871394] Lustre: Skipped 1 previous similar message [ 6285.043145] Lustre: lustre-MDT0001: new disk, initializing [ 6285.301048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6285.324137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6290.598783] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6301.511553] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6307.256819] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6315.413490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6315.916212] Lustre: lustre-OST0000: new disk, initializing [ 6315.919362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6315.923901] Lustre: 210184:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6323.314470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6323.323702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6323.480260] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6323.512272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6334.081290] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6339.364731] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6339.646308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6344.817155] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6345.007566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6349.280479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6349.286380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.291868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6349.300148] Lustre: Skipped 5 previous similar messages [ 6349.317455] Lustre: Skipped 8 previous similar messages [ 6361.055257] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6361.306520] Lustre: server umount lustre-OST0000 complete [ 6367.200427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6372.072214] Lustre: server umount lustre-MDT0000 complete [ 6374.887439] LustreError: 208228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6374.911526] LustreError: 208228:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6376.180173] LustreError: 209010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740732 with bad export cookie 4154409780982775819 [ 6376.199517] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6376.698598] Lustre: server umount lustre-MDT0001 complete [ 6385.126149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.674914] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6401.543385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.735037] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6414.978278] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6422.331937] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6430.604071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6437.658848] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6447.076218] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6452.606654] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6456.214001] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6461.859433] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6462.052099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6466.021176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6466.025753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6466.051799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6466.057632] Lustre: Skipped 3 previous similar messages [ 6466.070474] Lustre: Skipped 8 previous similar messages [ 6470.848204] Lustre: server umount lustre-OST0000 complete [ 6478.945876] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6484.977735] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6514.410614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.159189] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6532.137513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.148261] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.151803] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6532.180128] Lustre: Skipped 1 previous similar message [ 6533.713118] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6536.313413] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6542.305892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6542.321713] LustreError: Skipped 1 previous similar message [ 6547.685708] Lustre: server umount lustre-OST0000 complete [ 6557.993042] Lustre: server umount lustre-MDT0000 complete [ 6561.960345] LustreError: 212385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790740917 with bad export cookie 4154409780982776701 [ 6561.962448] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6561.968879] LustreError: 212385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6562.762213] Lustre: server umount lustre-MDT0001 complete [ 6571.740628] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:02:06 (1790740926) [ 6578.224028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6578.899949] LustreError: 218412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.923739] LustreError: 218412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6578.988904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6579.001744] Lustre: Skipped 6 previous similar messages [ 6584.110342] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6593.122451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.264666] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6605.533551] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6611.737707] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6618.922300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6625.736080] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6634.037906] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6639.887902] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6640.199767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6645.501647] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6645.704748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6652.890355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6653.150178] Lustre: lustre-OST0001: new disk, initializing [ 6653.153893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6653.161916] Lustre: 221781:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6660.018773] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6661.744394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6661.751852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6661.859314] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6669.987053] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6675.662118] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6675.981651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6680.264299] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6680.482147] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6682.600137] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6682.602728] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.616242] LustreError: Skipped 1 previous similar message [ 6682.629816] Lustre: Skipped 4 previous similar messages [ 6682.636144] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6682.636153] Lustre: Skipped 8 previous similar messages [ 6688.448974] Lustre: server umount lustre-OST0001 complete [ 6696.474845] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6696.713382] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6701.783709] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6702.016521] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6731.656375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6737.383142] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6737.411932] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6737.424585] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6737.455487] Lustre: Skipped 1 previous similar message [ 6739.151435] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6747.144367] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6748.752828] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6757.418247] Lustre: server umount lustre-OST0001 complete [ 6780.878577] Lustre: server umount lustre-OST0000 complete [ 6791.579956] Lustre: server umount lustre-MDT0000 complete [ 6796.194782] LustreError: 218393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741152 with bad export cookie 4154409780982777667 [ 6796.201212] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6796.203508] LustreError: 218393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6802.995462] Lustre: server umount lustre-MDT0001 complete [ 6829.195288] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 00:06:23 (1790741183) [ 6835.583205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.822532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6835.859899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6841.050193] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6850.363473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.452970] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6850.491526] Lustre: 227031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6855.606242] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6863.022946] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6868.137631] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6874.410468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6874.580672] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6882.205406] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6890.815926] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6900.792655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6900.840536] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6908.236540] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 6916.809757] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6918.445280] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6922.900373] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6923.143472] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6928.468601] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6928.673828] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6935.803771] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94d391627800.ost_server_uuid 50 [ 6937.757729] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff94d391627800.ost_server_uuid in IDLE state after 0 sec [ 6939.133219] Lustre: Permanently deactivating lustre-OST0001 [ 6939.162373] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6943.107798] Lustre: Permanently reactivating lustre-OST0001 [ 6948.839506] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6948.845516] LustreError: Skipped 5 previous similar messages [ 6948.857473] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.877287] Lustre: Skipped 6 previous similar messages [ 6948.889762] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6948.895038] Lustre: Skipped 11 previous similar messages [ 6962.143355] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6962.288264] Lustre: server umount lustre-OST0001 complete [ 6962.674163] LustreError: 228874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.708102] LustreError: 228874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 37 previous similar messages [ 6972.511692] Lustre: server umount lustre-OST0000 complete [ 6982.442057] Lustre: server umount lustre-MDT0000 complete [ 6986.151821] LustreError: 227785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741342 with bad export cookie 4154409780982778955 [ 6986.162811] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6986.180299] LustreError: 227785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6986.747651] Lustre: server umount lustre-MDT0001 complete [ 7014.510093] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:09:28 (1790741368) [ 7024.465365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.971901] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7025.062348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7025.079787] Lustre: Skipped 5 previous similar messages [ 7029.039335] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7038.789532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.879162] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7038.908373] Lustre: 233728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7045.120336] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7053.824973] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7060.053773] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7068.142683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.346249] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7068.377394] Lustre: Permanently deactivating lustre-OST0000 [ 7068.385892] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7075.544847] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7077.521961] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7077.544897] LustreError: 236034:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 7083.803603] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7092.922132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7093.031685] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7093.287039] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7093.299636] Lustre: Skipped 11 previous similar messages [ 7099.670131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7101.569502] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7111.677224] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.453499] Lustre: Permanently reactivating lustre-OST0000 [ 7113.466433] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7113.471619] Lustre: Skipped 2 previous similar messages [ 7123.560952] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7123.563295] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 46 seconds. I think it's dead, and I am evicting it. exp 00000000c751e2e9, cur 1790741479 deadline 1790741463 last 1790741433 [ 7123.570628] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7123.591113] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7123.699387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7134.536397] Lustre: server umount lustre-OST0001 complete [ 7144.438736] Lustre: server umount lustre-OST0000 complete [ 7163.359473] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7163.665650] Lustre: server umount lustre-MDT0000 complete [ 7167.469199] LustreError: 233719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741523 with bad export cookie 4154409780982780593 [ 7167.474877] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7167.488821] LustreError: 233719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7167.951842] Lustre: server umount lustre-MDT0001 complete [ 7179.431155] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 7182.841527] Key type lgssc unregistered [ 7183.203696] LNet: 238612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7183.211146] LNetError: 238612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7183.242536] LNet: Removed LNI 192.168.201.135@tcp [ 7184.354057] Key type .llcrypt unregistered [ 7184.356686] Key type ._llcrypt unregistered [ 7209.185530] Key type ._llcrypt registered [ 7209.188180] Key type .llcrypt registered [ 7209.358508] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 7211.268690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7211.296253] alg: No test for adler32 (adler32-zlib) [ 7212.460670] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 7212.697650] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 7214.391280] Key type lgssc registered [ 7215.949298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7229.294472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7229.334845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.932244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7236.156919] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7240.161793] LustreError: 240099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.214154] LustreError: 240099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7245.282494] LustreError: 240100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.202658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.432319] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7246.508772] Lustre: 240088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7246.527776] Lustre: Permanently deactivating lustre-MDT0001 [ 7246.560434] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7247.186467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7253.419810] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7254.597675] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7263.595788] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7270.797934] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7277.976659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7278.412676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7285.621666] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7291.715827] LustreError: 241943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.122030] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7297.008250] LustreError: 241941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.117406] LustreError: 241944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.135634] LustreError: 241944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7301.148216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7301.640448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7301.840969] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7307.258952] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7307.261873] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7309.177674] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7318.345138] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7322.879330] Lustre: Permanently reactivating lustre-MDT0001 [ 7322.914769] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7322.922736] Lustre: Skipped 2 previous similar messages [ 7324.541826] Lustre: lustre-MDT0001: Client d9302b53-73a8-4888-ac89-3e649fda209a (at 192.168.201.35@tcp) reconnecting [ 7332.504182] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.540094] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7332.554348] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7332.575975] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7332.678694] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7334.277149] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7341.013090] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7341.309660] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7346.391992] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7346.683477] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7349.736481] Lustre: Permanently deactivating lustre-MDT0001 [ 7349.758672] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7349.766782] Lustre: Skipped 2 previous similar messages [ 7355.506509] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7365.092523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7365.102274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7365.123922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.101217] Lustre: server umount lustre-MDT0000 complete [ 7368.679267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.682749] LustreError: 241361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7368.696735] Lustre: Skipped 2 previous similar messages [ 7368.719079] LustreError: 241361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7370.161041] LustreError: 243651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741726 with bad export cookie 13092291681456916938 [ 7370.178908] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.179650] LustreError: 243651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7370.465395] Lustre: server umount lustre-MDT0001 complete [ 7376.886128] Lustre: server umount lustre-OST0000 complete [ 7388.280180] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:15:42 (1790741742) [ 7400.292982] Lustre: server umount lustre-OST0001 complete [ 7408.678154] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 7420.149286] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 7461.479975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.695548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7461.705324] Lustre: Skipped 2 previous similar messages [ 7461.756450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7461.837439] Lustre: lustre-MDT0000: new disk, initializing [ 7461.890959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.905201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7466.625693] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7477.051899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.176218] Lustre: 248361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7477.231051] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7477.236971] Lustre: Skipped 1 previous similar message [ 7477.330687] Lustre: lustre-MDT0001: new disk, initializing [ 7477.487555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7477.528244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7477.542708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7483.224640] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7492.257534] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7497.709302] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7503.967100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.262938] Lustre: lustre-OST0000: new disk, initializing [ 7504.267179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7504.273547] Lustre: 250331:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7504.384052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7510.872831] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7513.168541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7513.187848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7513.320582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7521.234686] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7523.054798] LustreError: 248368:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7532.378614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7532.638228] Lustre: lustre-OST0001: new disk, initializing [ 7532.647189] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7532.655896] Lustre: 251447:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7533.079187] LustreError: 248368:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7533.108780] LustreError: 248369:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7539.804865] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7542.422573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7542.442893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7542.604152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7543.159959] LustreError: 248369:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7549.629962] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7550.828467] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7552.996316] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7553.000853] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7553.045567] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7553.060784] Lustre: Skipped 1 previous similar message [ 7558.116654] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7558.131728] Lustre: Skipped 1 previous similar message [ 7558.888584] Lustre: server umount lustre-OST0001 complete [ 7563.233349] LustreError: 251903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.279810] LustreError: 251903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7578.613857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.625390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7578.632988] Lustre: Skipped 2 previous similar messages [ 7578.639955] Lustre: Skipped 1 previous similar message [ 7582.964643] Lustre: server umount lustre-OST0000 complete [ 7588.836771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7588.845887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7588.858419] Lustre: Skipped 1 previous similar message [ 7588.873540] Lustre: Skipped 1 previous similar message [ 7593.363272] Lustre: server umount lustre-MDT0000 complete [ 7597.204534] LustreError: 248352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790741953 with bad export cookie 13092291681456919535 [ 7597.215856] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7597.224564] LustreError: 248352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7603.671951] Lustre: server umount lustre-MDT0001 complete [ 7615.006601] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 7617.885480] Key type lgssc unregistered [ 7618.225569] LNet: 253478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7618.232422] LNetError: 253478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7618.257901] LNet: Removed LNI 192.168.201.135@tcp [ 7619.146125] Key type .llcrypt unregistered [ 7619.150948] Key type ._llcrypt unregistered [ 7655.370585] Key type ._llcrypt registered [ 7655.375365] Key type .llcrypt registered [ 7655.559770] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 7657.491620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7657.518517] alg: No test for adler32 (adler32-zlib) [ 7658.730760] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 7659.073588] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 7660.769908] Key type lgssc registered [ 7662.274987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7674.356993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7674.395495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.795559] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7675.826441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7675.945792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.550235] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7690.593338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.802261] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7690.863921] Lustre: 255698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7691.233631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7696.504830] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7704.520178] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7709.801382] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7716.049134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.197376] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7716.445481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.185914] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7725.233096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7733.324598] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7786.466523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7786.477049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.492194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7786.979539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7786.985696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7787.005038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7792.419508] Lustre: server umount lustre-OST0000 complete [ 7799.538772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7816.314846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7816.738485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7817.778237] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7817.803436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7817.821193] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7824.109493] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7832.405690] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7874.023462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.045744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7874.060296] Lustre: Skipped 1 previous similar message [ 7879.155729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7879.170588] Lustre: Skipped 1 previous similar message [ 7879.709810] Lustre: server umount lustre-OST0000 complete [ 7884.259896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.263237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7884.283540] Lustre: Skipped 2 previous similar messages [ 7884.296579] Lustre: Skipped 1 previous similar message [ 7890.351430] Lustre: server umount lustre-MDT0000 complete [ 7894.496813] LustreError: 255709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7894.538117] LustreError: 255709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7894.586263] LustreError: 255690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742250 with bad export cookie 10054273842349099632 [ 7894.599371] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7894.602229] LustreError: 255690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7895.073773] Lustre: server umount lustre-MDT0001 complete [ 7905.958788] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 7908.701146] Key type lgssc unregistered [ 7909.036441] LNet: 261122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7909.040807] LNetError: 261122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7910.053594] LNet: Removed LNI 192.168.201.135@tcp [ 7910.920843] Key type .llcrypt unregistered [ 7910.923706] Key type ._llcrypt unregistered [ 7940.352446] Key type ._llcrypt registered [ 7940.354578] Key type .llcrypt registered [ 7940.454042] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 7941.908712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7942.024462] alg: No test for adler32 (adler32-zlib) [ 7943.289640] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 7943.596567] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 7945.319239] Key type lgssc registered [ 7946.749613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7954.751425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7954.766272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.424112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.775456] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7966.692160] LustreError: 262557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.714715] LustreError: 262557:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7970.938776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7971.490656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7976.480662] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 7983.857878] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7989.183812] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7998.335872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.905844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8001.982185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8006.329616] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8015.651927] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8042.978990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8042.993225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8043.020628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8047.075511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8047.086077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8047.090725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8047.113103] Lustre: Skipped 1 previous similar message [ 8049.124434] Lustre: server umount lustre-OST0000 complete [ 8057.314601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.324012] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8057.342498] Lustre: Skipped 1 previous similar message [ 8057.353578] Lustre: Skipped 1 previous similar message [ 8059.928284] Lustre: server umount lustre-MDT0000 complete [ 8062.434155] LustreError: 265071:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8062.452475] LustreError: 265071:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8063.885058] LustreError: 262540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742419 with bad export cookie 18004120517450439874 [ 8063.911455] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8063.926895] LustreError: 262540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8064.724046] Lustre: server umount lustre-MDT0001 complete [ 8077.741978] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 8080.495718] Key type lgssc unregistered [ 8080.841180] LNet: 266864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8080.848674] LNetError: 266864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8080.885912] LNet: Removed LNI 192.168.201.135@tcp [ 8082.044716] Key type .llcrypt unregistered [ 8082.047911] Key type ._llcrypt unregistered [ 8105.476501] Key type ._llcrypt registered [ 8105.479936] Key type .llcrypt registered [ 8105.648451] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 8107.679775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8107.688340] alg: No test for adler32 (adler32-zlib) [ 8108.902556] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 8109.343876] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 8111.127838] Key type lgssc registered [ 8112.941428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8121.837238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8121.882758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8123.376721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.180966] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8133.612655] LustreError: 268103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8133.627857] LustreError: 268103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8134.632164] LustreError: 268102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8139.762350] LustreError: 268103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8140.111334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.761642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8146.868450] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8156.544800] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8162.743792] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8171.413181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8171.805629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.243748] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8189.469368] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8194.363948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8209.897052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.907951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8209.916835] Lustre: Skipped 1 previous similar message [ 8209.920274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8209.941218] Lustre: Skipped 1 previous similar message [ 8215.008898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8215.028976] Lustre: Skipped 1 previous similar message [ 8220.129963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8220.137503] Lustre: Skipped 1 previous similar message [ 8221.151217] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8221.361964] Lustre: server umount lustre-OST0000 complete [ 8227.808510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8227.816623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8227.837917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8230.369864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8232.424224] Lustre: server umount lustre-MDT0000 complete [ 8235.497364] LustreError: 268099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8235.518177] LustreError: 268099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8236.742790] LustreError: 268082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742592 with bad export cookie 3383111640938672650 [ 8236.752989] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8236.765355] LustreError: 268082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8237.235501] Lustre: server umount lustre-MDT0001 complete [ 8250.635866] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 8254.451173] Key type lgssc unregistered [ 8254.739464] LNet: 271785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8254.748626] LNetError: 271785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8254.763601] LNet: Removed LNI 192.168.201.135@tcp [ 8255.705218] Key type .llcrypt unregistered [ 8255.707923] Key type ._llcrypt unregistered [ 8282.709387] Key type ._llcrypt registered [ 8282.715224] Key type .llcrypt registered [ 8282.822230] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 8284.158789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8284.328600] alg: No test for adler32 (adler32-zlib) [ 8285.687950] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 8286.157490] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 8287.959448] Key type lgssc registered [ 8289.604748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8298.568784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8298.579477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.123704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8306.523558] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8309.219267] LustreError: 273221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8309.237869] LustreError: 273221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8314.346256] LustreError: 273220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8316.222712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8316.708919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8322.183253] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8330.334682] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8336.954321] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8344.503334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8344.911285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8351.447361] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8359.427373] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8369.468726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8395.235288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8395.258485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8400.434369] Lustre: server umount lustre-OST0000 complete [ 8403.949924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8403.966447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.984951] Lustre: Skipped 1 previous similar message [ 8403.997908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8404.011573] Lustre: Skipped 1 previous similar message [ 8405.473485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8405.479175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8410.537829] Lustre: server umount lustre-MDT0000 complete [ 8410.598125] LustreError: 273950:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.617698] LustreError: 273950:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8414.421928] LustreError: 273202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742770 with bad export cookie 5450809264598435177 [ 8414.428602] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8414.807435] Lustre: server umount lustre-MDT0001 complete [ 8426.353397] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 8430.362165] Key type lgssc unregistered [ 8430.862130] LNet: 277550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8430.884145] LNetError: 277550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8431.920418] LNet: Removed LNI 192.168.201.135@tcp [ 8432.768167] Key type .llcrypt unregistered [ 8432.770346] Key type ._llcrypt unregistered [ 8457.437330] Key type ._llcrypt registered [ 8457.442716] Key type .llcrypt registered [ 8457.652202] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 8459.535684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8459.607071] alg: No test for adler32 (adler32-zlib) [ 8461.214935] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 8461.549258] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 8463.375279] Key type lgssc registered [ 8464.906647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8473.066762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8473.109158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8474.910909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8481.305820] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8484.327080] LustreError: 278806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8484.348762] LustreError: 278806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8489.447244] LustreError: 278808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.806220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.118354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8495.694907] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8502.688405] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8508.736300] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8517.491168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8517.787677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8519.860370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8524.447160] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8534.139277] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8550.893502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8550.896430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.902938] LustreError: Skipped 1 previous similar message [ 8550.932747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8550.961036] Lustre: Skipped 1 previous similar message [ 8553.848515] Lustre: server umount lustre-OST0000 complete [ 8561.120875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.141435] Lustre: Skipped 1 previous similar message [ 8561.144618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.154515] Lustre: Skipped 1 previous similar message [ 8564.133418] Lustre: server umount lustre-MDT0000 complete [ 8566.248518] LustreError: 278814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8566.273193] LustreError: 278814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8568.072878] LustreError: 278771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790742923 with bad export cookie 16478866723188363455 [ 8568.091840] LustreError: 278771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8568.092674] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8568.870748] Lustre: server umount lustre-MDT0001 complete [ 8582.901798] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 8586.165568] Key type lgssc unregistered [ 8586.669905] LNet: 282510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8586.678901] LNetError: 282510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8586.735278] LNet: Removed LNI 192.168.201.135@tcp [ 8587.853369] Key type .llcrypt unregistered [ 8587.857757] Key type ._llcrypt unregistered [ 8601.508358] Key type ._llcrypt registered [ 8601.512976] Key type .llcrypt registered [ 8601.645742] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8619.110483] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 8638.589377] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 8639.792352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8639.854031] alg: No test for adler32 (adler32-zlib) [ 8641.192390] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 8641.419556] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 8643.055242] Key type lgssc registered [ 8644.571387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8682.611909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8682.656453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8684.053110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8684.086591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8684.160812] Lustre: lustre-MDT0000: new disk, initializing [ 8684.310601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8684.361791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8689.174247] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8701.828693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.008678] Lustre: 286083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8702.112817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8702.120208] Lustre: Skipped 1 previous similar message [ 8702.235731] Lustre: lustre-MDT0001: new disk, initializing [ 8702.433936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8702.491909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8702.504461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8707.349617] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8717.097962] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8723.170537] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8729.488802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8729.799649] Lustre: lustre-OST0000: new disk, initializing [ 8729.804364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8729.815655] Lustre: 288049:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8729.912871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.006517] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8737.978877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8737.998108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8738.181801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8747.456292] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8752.534617] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8752.773527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8757.380314] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8757.562982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8763.900072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8763.903401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8763.911319] LustreError: Skipped 1 previous similar message [ 8763.916154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8763.930114] Lustre: Skipped 1 previous similar message [ 8763.964929] Lustre: Skipped 1 previous similar message [ 8766.027547] Lustre: server umount lustre-OST0000 complete [ 8774.117311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.141405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8774.150687] Lustre: Skipped 1 previous similar message [ 8779.235077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8779.241709] Lustre: Skipped 1 previous similar message [ 8783.330783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8783.342690] Lustre: Skipped 1 previous similar message [ 8784.355276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8784.675212] Lustre: server umount lustre-MDT0000 complete [ 8789.293794] LustreError: 286073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743145 with bad export cookie 16674848228692328704 [ 8789.302351] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.308167] LustreError: 286073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8789.755734] Lustre: server umount lustre-MDT0001 complete [ 8799.342078] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:39:13 (1790743153) [ 8807.653092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8808.182223] LustreError: 290465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.206631] LustreError: 290465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8808.270593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8813.405566] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8813.553301] LustreError: 290466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.665900] LustreError: 290465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8823.058416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8823.405929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8828.906800] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8836.327658] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8842.984885] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8850.880154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8851.264631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.254112] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 8866.193297] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8879.073501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8879.076642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8879.083871] Lustre: Skipped 1 previous similar message [ 8879.089945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8879.101462] LustreError: Skipped 1 previous similar message [ 8884.523150] Lustre: server umount lustre-OST0000 complete [ 8889.318537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8889.333146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8889.334572] Lustre: Skipped 2 previous similar messages [ 8889.353649] Lustre: Skipped 4 previous similar messages [ 8894.433419] LustreError: 290466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8894.444789] LustreError: 290466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8894.477245] Lustre: server umount lustre-MDT0000 complete [ 8898.208945] LustreError: 290446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743254 with bad export cookie 16674848228692329586 [ 8898.224147] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8898.230313] LustreError: 290446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8898.696826] Lustre: server umount lustre-MDT0001 complete [ 8909.523578] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 8912.630526] Key type lgssc unregistered [ 8912.936680] LNet: 294012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8912.945963] LNetError: 294012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8912.957300] LNet: Removed LNI 192.168.201.135@tcp [ 8913.775102] Key type .llcrypt unregistered [ 8913.777252] Key type ._llcrypt unregistered [ 8942.548908] Key type ._llcrypt registered [ 8942.551441] Key type .llcrypt registered [ 8942.663699] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 8959.181847] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 8960.506886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8960.628137] alg: No test for adler32 (adler32-zlib) [ 8961.837871] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 8962.042687] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 8963.719276] Key type lgssc registered [ 8964.922076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9009.175128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9009.218077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9010.666874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9010.697420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9010.821847] Lustre: lustre-MDT0000: new disk, initializing [ 9010.922913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9010.943148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9016.434188] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9027.098081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9027.240685] Lustre: 297853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9027.302928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9027.307127] Lustre: Skipped 1 previous similar message [ 9027.395811] Lustre: lustre-MDT0001: new disk, initializing [ 9027.532234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9027.610700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9027.617557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9032.954588] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9043.462085] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9050.486728] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9057.748902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9058.233787] Lustre: lustre-OST2710: new disk, initializing [ 9058.243019] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9058.263045] Lustre: 299823:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9058.419946] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9065.337079] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9066.605290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9066.627842] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9066.714495] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9075.480188] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9081.800654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9081.897133] Lustre: lustre-OST03e8: new disk, initializing [ 9081.899703] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9081.904682] Lustre: 300886:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9081.988387] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9088.795787] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9090.071336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9090.076524] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9090.183970] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9098.369616] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9107.334157] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9107.765674] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9113.536151] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9113.789303] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9118.572084] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9118.704865] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9124.159422] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9124.561513] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9129.956491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9129.965191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9129.990601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9130.983504] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9131.003814] Lustre: Skipped 2 previous similar messages [ 9131.092354] Lustre: server umount lustre-MDT0000 complete [ 9134.835590] LustreError: 297845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743490 with bad export cookie 10287017563082680149 [ 9134.842250] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9134.851758] LustreError: 297845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9135.217413] Lustre: server umount lustre-MDT0001 complete [ 9149.195791] Lustre: server umount lustre-OST2710 complete [ 9151.967175] Lustre: 295642:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790743491/real 1790743491] req@000000004f51c4d0 x1877730464436608/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790743507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9152.017827] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.973178] Lustre: server umount lustre-OST03e8 complete [ 9176.307566] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [ 9187.168973] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 9227.555537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9227.862994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9227.898616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9227.969919] Lustre: lustre-MDT0000: new disk, initializing [ 9228.118683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9228.153671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9233.127686] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9243.654694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9243.721403] Lustre: 306137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9243.754621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9243.756299] Lustre: Skipped 1 previous similar message [ 9243.796729] Lustre: lustre-MDT0001: new disk, initializing [ 9243.846431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9243.863047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9243.868305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9248.466326] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9258.884292] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9266.235960] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9274.434554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9274.613306] Lustre: lustre-OST0000: new disk, initializing [ 9274.616243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9274.620779] Lustre: 308104:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9274.701882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9281.260644] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9283.144623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9283.151657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9283.210417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9290.970118] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9297.777267] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9298.098986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9304.054100] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9304.438612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9309.165916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9309.175251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.192575] LustreError: Skipped 1 previous similar message [ 9309.207199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9309.211746] Lustre: Skipped 1 previous similar message [ 9309.233132] Lustre: Skipped 1 previous similar message [ 9313.325204] Lustre: server umount lustre-OST0000 complete [ 9319.396319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.416370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9319.424919] Lustre: Skipped 1 previous similar message [ 9324.514338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9324.527475] Lustre: Skipped 1 previous similar message [ 9324.729156] Lustre: server umount lustre-MDT0000 complete [ 9328.693731] LustreError: 306129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743684 with bad export cookie 10287017563082681829 [ 9328.700165] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9328.704078] LustreError: 306129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9329.247868] Lustre: server umount lustre-MDT0001 complete [ 9338.502346] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:48:13 (1790743693) [ 9340.288216] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9342.321658] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:48:16 (1790743696) [ 9346.692276] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 9374.262364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9374.500780] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9374.513829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9374.846591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.770059] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9390.931876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9391.028575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9391.069365] Lustre: 311962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9397.004981] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9419.506314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9419.759631] Lustre: Denying initial registration attempt from nid 192.168.201.135@tcp, specified as failover [ 9419.767588] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9419.779515] LustreError: 313417:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9419.788806] LustreError: 313417:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9419.795594] LustreError: 313417:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9419.868831] Lustre: server umount lustre-OST0000 complete [ 9419.876534] LustreError: 313417:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9427.425385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9427.430554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.438423] Lustre: Skipped 1 previous similar message [ 9427.455267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9433.731129] Lustre: server umount lustre-MDT0000 complete [ 9437.902496] LustreError: 311954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743793 with bad export cookie 10287017563082682711 [ 9437.906631] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9437.920760] LustreError: 311954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9438.526295] Lustre: server umount lustre-MDT0001 complete [ 9452.458204] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 9456.115332] Key type lgssc unregistered [ 9456.488550] LNet: 314539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9456.498891] LNetError: 314539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9456.517686] LNet: Removed LNI 192.168.201.135@tcp [ 9457.489982] Key type .llcrypt unregistered [ 9457.493044] Key type ._llcrypt unregistered [ 9471.215814] Key type ._llcrypt registered [ 9471.222442] Key type .llcrypt registered [ 9471.321517] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 9472.464900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9472.517332] alg: No test for adler32 (adler32-zlib) [ 9473.639816] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 9473.869445] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 9475.559286] Key type lgssc registered [ 9477.369507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9527.083946] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 9537.606842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9537.645404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9538.920387] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9538.939511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9539.094806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9543.763983] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9552.695831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.810720] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9552.853063] Lustre: 317321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9553.124987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9557.615970] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9564.660917] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9570.413874] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9590.715135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.986107] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9591.259869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9597.257045] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9603.997144] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9607.168172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9607.177461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.205500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9612.140906] Lustre: server umount lustre-OST0000 complete [ 9617.376483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.401085] Lustre: Skipped 1 previous similar message [ 9617.410953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9617.423035] Lustre: Skipped 2 previous similar messages [ 9622.504863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9622.519658] Lustre: Skipped 1 previous similar message [ 9622.660490] Lustre: server umount lustre-MDT0000 complete [ 9626.549854] LustreError: 317314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790743982 with bad export cookie 3202028324990564504 [ 9626.557871] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9626.562736] LustreError: 317314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9627.038907] Lustre: server umount lustre-MDT0001 complete [ 9639.486303] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 9642.610747] Key type lgssc unregistered [ 9642.855384] LNet: 320896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9642.864484] LNetError: 320896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9643.886220] LNet: Removed LNI 192.168.201.135@tcp [ 9644.842809] Key type .llcrypt unregistered [ 9644.848927] Key type ._llcrypt unregistered [ 9671.838672] Key type ._llcrypt registered [ 9671.848806] Key type .llcrypt registered [ 9672.004781] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 9673.682194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9673.918564] alg: No test for adler32 (adler32-zlib) [ 9675.496875] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 9675.900044] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 9677.703223] Key type lgssc registered [ 9680.424136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9690.574534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9690.601038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9692.193806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9697.532357] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9701.357905] LustreError: 322327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9701.394055] LustreError: 322327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9706.463926] LustreError: 322328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.847382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.282385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9712.426403] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9721.023563] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9727.234103] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9737.140385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9737.623220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9744.137799] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9752.243362] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9771.492806] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9771.500669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9771.510722] Lustre: Skipped 2 previous similar messages [ 9771.516707] Lustre: Skipped 1 previous similar message [ 9773.338055] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.35@tcp (stopping) [ 9773.346731] Lustre: Skipped 1 previous similar message [ 9775.846989] Lustre: server umount lustre-MDT0000 complete [ 9776.610153] LustreError: 322327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9776.647736] LustreError: 322327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9780.692199] LustreError: 322307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744136 with bad export cookie 13072067747405071902 [ 9780.702192] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9780.703688] LustreError: 322307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9781.732877] LustreError: 322322:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9781.734970] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9781.757444] LustreError: 322322:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9781.781481] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9785.760201] LustreError: 322324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9785.760470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9785.796427] LustreError: 322324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9785.829238] Lustre: Skipped 1 previous similar message [ 9790.889821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9790.900535] Lustre: Skipped 1 previous similar message [ 9793.798680] LustreError: 322322:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9793.816089] LustreError: 322322:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9795.039140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9795.465054] Lustre: server umount lustre-MDT0001 complete [ 9802.851367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9811.975675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9826.168160] LustreError: 324161:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9826.186637] LustreError: 324161:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9826.274533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9826.350884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:257) [ 9831.068301] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9840.466595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9840.929400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9846.067394] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9846.281325] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9854.636198] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9856.195413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9862.526890] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9864.019355] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9871.844523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9871.848178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9871.853416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9871.861366] Lustre: Skipped 3 previous similar messages [ 9884.127239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9884.485926] Lustre: server umount lustre-OST0000 complete [ 9892.323252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9892.357348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9892.368828] Lustre: Skipped 1 previous similar message [ 9892.372061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9892.411833] Lustre: Skipped 5 previous similar messages [ 9896.269405] Lustre: server umount lustre-MDT0000 complete [ 9897.447990] LustreError: 325950:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.490613] LustreError: 325950:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9900.823051] LustreError: 322307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744256 with bad export cookie 13072067747405079588 [ 9900.849354] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9901.250425] Lustre: server umount lustre-MDT0001 complete [ 9912.298444] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [ 9915.997913] Key type lgssc unregistered [ 9916.201817] LNet: 328621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9916.212187] LNetError: 328621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9916.242963] LNet: Removed LNI 192.168.201.135@tcp [ 9917.146263] Key type .llcrypt unregistered [ 9917.148257] Key type ._llcrypt unregistered [ 9944.170853] Key type ._llcrypt registered [ 9944.172711] Key type .llcrypt registered [ 9944.347827] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [ 9945.636373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9945.658912] alg: No test for adler32 (adler32-zlib) [ 9946.759512] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [ 9947.010260] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [ 9948.672324] Key type lgssc registered [ 9949.805205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9957.856864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9957.871240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.187143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9963.149871] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9969.634068] LustreError: 330052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.649394] LustreError: 330052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9971.263532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.604149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9976.157880] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [ 9985.402466] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9992.007895] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9996.518976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.541354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9996.549392] Lustre: Skipped 1 previous similar message [10001.888452] LustreError: 330047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10001.891611] Lustre: server umount lustre-MDT0000 complete [10001.900229] LustreError: 330047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10006.116903] LustreError: 330034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744361 with bad export cookie 5024259419008065085 [10006.122621] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10006.129478] LustreError: 330034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10006.829994] Lustre: server umount lustre-MDT0001 complete [10022.952185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.426827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10023.487872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10023.678243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10029.018156] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10037.604620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10037.684834] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10037.716955] Lustre: 332641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10038.068965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10043.641302] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10052.064740] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10057.913847] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10068.236944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10068.566929] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10068.579358] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10068.592412] LustreError: 334525:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10068.600303] LustreError: 334525:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10068.609457] LustreError: 334525:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10068.744693] Lustre: server umount lustre-OST0000 complete [10068.746794] LustreError: 334525:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10080.142510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10080.468707] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10080.707540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10087.562352] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10088.530896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:289) [10094.351128] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10102.881984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10103.029113] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10103.161501] Lustre: lustre-OST0001: new disk, initializing [10103.169892] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10103.182518] Lustre: 336068:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10103.307954] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10111.031938] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10113.226778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10113.253192] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10113.446964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10119.825546] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10123.747255] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10123.755610] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10123.766195] LustreError: Skipped 1 previous similar message [10123.785401] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10123.801742] Lustre: Skipped 2 previous similar messages [10123.815256] Lustre: Skipped 1 previous similar message [10128.869573] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10129.125980] Lustre: server umount lustre-OST0001 complete [10133.987053] LustreError: 334994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10133.987440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.000082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10134.000092] Lustre: Skipped 1 previous similar message [10134.010548] LustreError: 334994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10134.041642] Lustre: Skipped 1 previous similar message [10139.112880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10139.123602] Lustre: Skipped 2 previous similar messages [10139.127980] LustreError: 336530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10139.166858] LustreError: 336530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10139.901219] Lustre: server umount lustre-OST0000 complete [10144.230336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10144.249959] Lustre: Skipped 1 previous similar message [10149.348598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10149.364099] Lustre: Skipped 3 previous similar messages [10150.051887] Lustre: server umount lustre-MDT0000 complete [10153.689293] LustreError: 335009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744509 with bad export cookie 5024259419008065638 [10153.696237] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10153.700716] LustreError: 335009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10153.935312] Lustre: server umount lustre-MDT0001 complete [10164.970350] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [10168.653460] Key type lgssc unregistered [10169.046789] LNet: 338040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10169.051605] LNetError: 338040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10169.069301] LNet: Removed LNI 192.168.201.135@tcp [10170.151740] Key type .llcrypt unregistered [10170.153987] Key type ._llcrypt unregistered [10246.327590] Key type ._llcrypt registered [10246.330125] Key type .llcrypt registered [10246.555077] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [10264.123692] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [10265.717358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10265.736932] alg: No test for adler32 (adler32-zlib) [10267.209521] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [10267.596315] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [10269.407344] Key type lgssc registered [10271.293741] Lustre: Echo OBD driver; http://www.lustre.org/ [10309.065849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10309.101217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.452366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10310.502330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10310.590246] Lustre: lustre-MDT0000: new disk, initializing [10310.703285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10310.725106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10315.067602] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10325.637137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.778677] Lustre: 342987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10325.836885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10325.840829] Lustre: Skipped 1 previous similar message [10325.925328] Lustre: lustre-MDT0001: new disk, initializing [10326.101710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10326.132601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10326.141084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10330.792693] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10338.484894] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10343.297363] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10350.060747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10350.431675] Lustre: lustre-OST0000: new disk, initializing [10350.437743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10350.447790] Lustre: 344955:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10350.563648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10356.568720] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10360.414024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10360.432739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10360.629725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10366.944329] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10372.704482] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10373.189702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10379.225763] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10379.519967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10386.415869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10386.417754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10386.426150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10386.459580] Lustre: Skipped 1 previous similar message [10391.522957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10391.539608] Lustre: Skipped 2 previous similar messages [10396.127155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10396.255327] Lustre: server umount lustre-OST0000 complete [10401.762946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10401.783426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10401.800183] Lustre: Skipped 1 previous similar message [10406.880672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10406.903352] Lustre: Skipped 1 previous similar message [10407.817153] Lustre: server umount lustre-MDT0000 complete [10412.622433] LustreError: 344954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744768 with bad export cookie 17983705546092949584 [10412.642139] LustreError: 344954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10412.644962] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10413.114595] Lustre: server umount lustre-MDT0001 complete [10422.849847] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:06:17 (1790744777) [10430.810407] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:06:25 (1790744785) [10440.316304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.738460] LustreError: 347713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.762767] LustreError: 347713:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10440.839032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10445.473651] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10446.306336] LustreError: 347714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.415843] LustreError: 347713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10454.881622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10455.275657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10459.756962] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10466.221967] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10471.276794] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10477.749305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10477.994689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10484.697063] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10491.980756] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10500.757985] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10502.496814] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10504.534056] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10506.547737] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10508.677084] Lustre: Failing over lustre-MDT0000 [10508.866909] Lustre: server umount lustre-MDT0000 complete [10510.119224] LustreError: 350204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10510.820198] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.839385] Lustre: Skipped 2 previous similar messages [10515.197765] LustreError: 350204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10515.231572] LustreError: 350204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10525.455996] LustreError: 347709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.482217] LustreError: 347709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10526.196609] Lustre: 341200:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790744866/real 1790744866] req@00000000f6a32dbf x1877731833878656/t0(0) o400->MGC192.168.201.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790744882 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10526.243906] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10529.167847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10536.417410] LustreError: 341198:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004148352d x1877731833884160/t0(0) o250->MGC192.168.201.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10536.784783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10536.850823] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10541.825230] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10542.063129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10542.111937] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10542.141537] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10542.204319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10552.300058] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10553.852924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10557.592618] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10559.655302] Lustre: Failing over lustre-MDT0000 [10560.052611] Lustre: server umount lustre-MDT0000 complete [10562.370515] LustreError: 347710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10562.393121] LustreError: 347710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [10562.534266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10562.546401] LustreError: Skipped 1 previous similar message [10562.549752] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10562.573366] Lustre: Skipped 3 previous similar messages [10579.223797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10579.335567] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10579.682392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10579.753321] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10583.810141] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10584.958496] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10585.071342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10585.077271] Lustre: Skipped 2 previous similar messages [10585.122398] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10585.213398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10594.781325] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10596.445830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10605.545283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10605.552833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.557085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10605.564712] Lustre: Skipped 2 previous similar messages [10605.575238] Lustre: Skipped 2 previous similar messages [10609.136952] Lustre: server umount lustre-MDT0000 complete [10610.657157] LustreError: 347713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.675745] LustreError: 347713:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [10612.745957] LustreError: 347694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790744968 with bad export cookie 17983705546092952293 [10612.756038] LustreError: 347694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10612.763158] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10612.995539] Lustre: server umount lustre-MDT0001 complete [10627.229910] Lustre: server umount lustre-OST0000 complete [10647.883530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.903543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10655.459802] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10665.520321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10670.132351] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10678.014448] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10683.380528] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10689.505057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10694.706222] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10694.867643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10701.040103] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10705.376122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10705.380751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10705.381956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10705.385244] Lustre: Skipped 1 previous similar message [10705.390174] Lustre: Skipped 1 previous similar message [10710.846922] Lustre: server umount lustre-OST0000 complete [10721.531841] Lustre: server umount lustre-MDT0000 complete [10725.345357] LustreError: 354034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745081 with bad export cookie 17983705546092953042 [10725.353974] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10725.355434] LustreError: 354034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10725.739829] Lustre: server umount lustre-MDT0001 complete [10737.870847] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [10740.745664] Key type lgssc unregistered [10740.978627] LNet: 357586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10740.990080] LNetError: 357586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10741.001214] LNet: Removed LNI 192.168.201.135@tcp [10741.931206] Key type .llcrypt unregistered [10741.933314] Key type ._llcrypt unregistered [10768.449527] Key type ._llcrypt registered [10768.453278] Key type .llcrypt registered [10768.550948] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [10770.039944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10770.072200] alg: No test for adler32 (adler32-zlib) [10771.061245] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [10771.252243] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [10772.911242] Key type lgssc registered [10774.321295] Lustre: Echo OBD driver; http://www.lustre.org/ [10782.213753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10782.236148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.655266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10787.919646] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10793.957858] LustreError: 359119:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10793.977080] LustreError: 359119:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10796.189157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.468538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10800.687587] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10811.136360] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10817.801733] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10827.573496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10828.037933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10829.097124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10836.115911] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10845.418677] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10856.932769] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.937863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10856.955188] Lustre: Skipped 1 previous similar message [10856.978173] Lustre: Skipped 2 previous similar messages [10859.376088] Lustre: server umount lustre-MDT0000 complete [10862.063569] LustreError: 359118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.099237] LustreError: 359118:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10863.850181] LustreError: 359101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745219 with bad export cookie 8736391249915676022 [10863.861984] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10863.863137] LustreError: 359101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10864.445574] Lustre: server umount lustre-MDT0001 complete [10880.245337] Lustre: server umount lustre-OST0000 complete [10896.487057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10896.782655] LustreError: 362832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.844851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10901.798167] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10901.994449] LustreError: 362833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.114365] LustreError: 362832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10911.474294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.024728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10916.944550] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10925.202440] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10931.260469] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10938.829426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10939.085631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10941.170712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10946.540962] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [10954.147574] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10967.010676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10967.026151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10967.030366] Lustre: Skipped 2 previous similar messages [10967.063023] Lustre: Skipped 2 previous similar messages [10972.129791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10972.770780] Lustre: server umount lustre-MDT0000 complete [10976.510195] LustreError: 363567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745332 with bad export cookie 8736391249915677135 [10976.534404] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10976.534973] LustreError: 363567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10976.918732] Lustre: server umount lustre-MDT0001 complete [10991.440366] Lustre: server umount lustre-OST0000 complete [11008.016317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11008.690259] LustreError: 366744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.804537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11014.152671] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11023.524028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11028.737920] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11036.658044] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11042.901297] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11049.845331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11050.068395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11050.076791] Lustre: Skipped 1 previous similar message [11053.197657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11056.331972] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11063.454946] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11068.911834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11068.918760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.932109] Lustre: Skipped 1 previous similar message [11068.940101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11068.944800] Lustre: Skipped 2 previous similar messages [11070.448368] Lustre: server umount lustre-OST0000 complete [11075.041482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.045129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11075.057540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11075.057548] Lustre: Skipped 2 previous similar messages [11085.282824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11085.293554] Lustre: Skipped 4 previous similar messages [11088.351989] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11088.705747] Lustre: server umount lustre-MDT0000 complete [11090.402048] LustreError: 368122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.424260] LustreError: 368122:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11092.683317] LustreError: 366724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745448 with bad export cookie 8736391249915678157 [11092.688465] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11092.702047] LustreError: 366724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11093.211494] Lustre: server umount lustre-MDT0001 complete [11103.874657] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [11106.437026] Key type lgssc unregistered [11106.716644] LNet: 370278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11106.722040] LNetError: 370278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11107.766630] LNet: Removed LNI 192.168.201.135@tcp [11108.791558] Key type .llcrypt unregistered [11108.794119] Key type ._llcrypt unregistered [11133.439453] Key type ._llcrypt registered [11133.441361] Key type .llcrypt registered [11133.551534] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [11134.865972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11134.908882] alg: No test for adler32 (adler32-zlib) [11135.854624] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [11136.062693] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [11137.711320] Key type lgssc registered [11139.143617] Lustre: Echo OBD driver; http://www.lustre.org/ [11147.786534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11147.819840] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11147.829417] LustreError: 371735:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11147.872630] LustreError: 371735:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11147.881582] LustreError: 371735:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11147.885313] LustreError: 371735:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11147.890241] LustreError: 371735:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11157.716645] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11157.732118] LustreError: 372134:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11157.749269] LustreError: 372134:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11157.760441] LustreError: 372134:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11157.775792] LustreError: 372134:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11157.791253] LustreError: 372134:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11176.755221] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [11180.161692] Key type lgssc unregistered [11180.531606] LNet: 373136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11180.544971] LNetError: 373136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11180.557746] LNet: Removed LNI 192.168.201.135@tcp [11181.422865] Key type .llcrypt unregistered [11181.425802] Key type ._llcrypt unregistered [11198.962680] Key type ._llcrypt registered [11198.964785] Key type .llcrypt registered [11199.073180] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [11213.979650] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [11215.419659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11215.450908] alg: No test for adler32 (adler32-zlib) [11216.568522] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [11216.813786] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [11218.471208] Key type lgssc registered [11219.531035] Lustre: Echo OBD driver; http://www.lustre.org/ [11251.796269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11251.809782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.077488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11253.125923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11253.241858] Lustre: lustre-MDT0000: new disk, initializing [11253.404966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11253.442385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11259.907970] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11272.052766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11272.219149] Lustre: 376310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11272.319699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11272.330190] Lustre: Skipped 1 previous similar message [11272.468816] Lustre: lustre-MDT0001: new disk, initializing [11272.642130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11272.689152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11272.701759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11278.708552] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11289.510807] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11296.251229] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11303.511674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11303.854438] Lustre: lustre-OST0000: new disk, initializing [11303.861847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11303.873741] Lustre: 378277:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11304.003777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11310.830737] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11311.203672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11311.216440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11311.320935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11319.639735] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11325.103585] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11325.424961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11330.033538] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11330.292338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11337.203174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11337.209516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11337.214504] LustreError: Skipped 1 previous similar message [11337.220425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11337.231431] Lustre: Skipped 1 previous similar message [11337.251458] Lustre: Skipped 1 previous similar message [11338.553873] Lustre: server umount lustre-OST0000 complete [11342.305032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11342.325359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11342.331018] Lustre: Skipped 1 previous similar message [11347.426540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11347.447098] Lustre: Skipped 1 previous similar message [11348.555173] Lustre: server umount lustre-MDT0000 complete [11352.703950] LustreError: 376303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745708 with bad export cookie 3991775854839976267 [11352.708744] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11352.714514] LustreError: 376303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11353.112751] Lustre: server umount lustre-MDT0001 complete [11362.430347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11364.427792] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:21:58 (1790745718) [11372.044565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11372.634871] LustreError: 380740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11372.664787] LustreError: 380740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11372.801127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11377.489803] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11378.148974] LustreError: 380741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11382.246791] LustreError: 380740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11387.363554] LustreError: 380741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11387.563716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11388.069983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11394.238949] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11402.142585] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11409.930159] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11418.239703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11418.622228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11426.495205] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11433.554564] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11440.362569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11440.448435] Lustre: lustre-OST0001: new disk, initializing [11440.450097] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11440.452683] Lustre: 383576:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11440.500306] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11447.387896] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11450.481737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11450.489585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11450.659348] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11458.104752] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11465.002971] Lustre: lustre-OST0001: Not available for connect from 192.168.201.35@tcp (stopping) [11466.224217] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11466.225261] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11466.240961] LustreError: Skipped 1 previous similar message [11466.263855] Lustre: Skipped 2 previous similar messages [11468.089249] Lustre: server umount lustre-OST0001 complete [11470.088450] LustreError: 382561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.584964] LustreError: 382562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.611563] LustreError: 382562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11486.697543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11486.711978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11486.729069] Lustre: Skipped 1 previous similar message [11486.736760] Lustre: Skipped 2 previous similar messages [11492.209681] Lustre: server umount lustre-OST0000 complete [11500.512430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11500.519659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11500.538469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11500.540938] Lustre: Skipped 1 previous similar message [11503.854300] Lustre: server umount lustre-MDT0000 complete [11507.168677] LustreError: 380741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11507.219661] LustreError: 380741:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [11508.535171] LustreError: 380720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790745864 with bad export cookie 3991775854839977149 [11508.546371] LustreError: 380720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11508.547257] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11515.094623] Lustre: server umount lustre-MDT0001 complete [11528.955862] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [11532.683260] Key type lgssc unregistered [11533.125661] LNet: 385565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11533.141195] LNetError: 385565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11533.164165] LNet: Removed LNI 192.168.201.135@tcp [11534.310113] Key type .llcrypt unregistered [11534.311411] Key type ._llcrypt unregistered [11589.421906] Key type ._llcrypt registered [11589.428164] Key type .llcrypt registered [11589.561374] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [11590.759496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11590.943843] alg: No test for adler32 (adler32-zlib) [11592.023900] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [11592.282943] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [11593.959935] Key type lgssc registered [11595.221095] Lustre: Echo OBD driver; http://www.lustre.org/ [11602.855201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11602.891470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.218889] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11604.235438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11604.346388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11610.005894] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11618.204762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11618.330200] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11618.410804] Lustre: 388381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11618.658709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11623.844987] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11632.423526] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11639.605619] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11644.385313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11644.390059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11644.417519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11646.434775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11646.436404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11646.475170] Lustre: Skipped 1 previous similar message [11648.586850] Lustre: server umount lustre-MDT0000 complete [11652.667775] LustreError: 388375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746008 with bad export cookie 16649140035340258826 [11652.674434] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11652.684710] LustreError: 388375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11653.015278] Lustre: server umount lustre-MDT0001 complete [11663.218383] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:26:57 (1790746017) [11670.166128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11670.726240] LustreError: 390895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11670.740759] LustreError: 390895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11670.785138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11676.136252] LustreError: 390896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11677.734572] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11681.248771] LustreError: 390895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.376450] LustreError: 390896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.370240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11689.920610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11695.176927] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11703.704431] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11711.256897] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11719.304067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11719.525601] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11719.745639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11727.508597] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11735.939324] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11745.187921] Lustre: Permanently deactivating lustre-OST0000 [11745.203390] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11752.003342] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11753.636441] LustreError: 393835:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11753.645274] LustreError: 393835:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11753.651063] LustreError: 393835:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11757.393087] LustreError: 393884:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11757.400503] LustreError: 393884:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11765.794202] Lustre: server umount lustre-OST0000 complete [11768.933052] LustreError: 394135:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11768.939482] LustreError: 394135:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11771.872387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11771.884825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11771.907098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11773.926522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11773.929889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11773.950299] Lustre: Skipped 1 previous similar message [11779.055294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11779.065038] Lustre: Skipped 1 previous similar message [11782.193299] Lustre: server umount lustre-MDT0000 complete [11784.169363] LustreError: 391623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11784.186670] LustreError: 391623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11786.145992] LustreError: 390877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746142 with bad export cookie 16649140035340259407 [11786.149707] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11786.163473] LustreError: 390877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11796.601204] Lustre: server umount lustre-MDT0001 complete [11805.547959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11810.010330] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11813.302448] LustreError: 395295:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11819.614651] Lustre: 395634:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.135@tcp [11821.523435] Lustre: server umount MGS complete [11834.944360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11835.394580] LustreError: 396358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11835.416017] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11835.420162] Lustre: Skipped 1 previous similar message [11835.458818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11841.965759] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11853.839663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11854.222364] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11854.302829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11860.149229] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11867.998631] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11873.763617] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11875.598897] Lustre: Permanently reactivating lustre-OST0000 [11875.613896] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11875.618686] Lustre: Skipped 2 previous similar messages [11902.943211] LustreError: 398129:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11911.359954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11911.990325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11919.069117] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [11927.325764] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.665542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.671098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11933.672148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11933.691695] Lustre: Skipped 1 previous similar message [11933.720188] LustreError: Skipped 1 previous similar message [11934.409403] Lustre: server umount lustre-OST0000 complete [11941.345037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11943.907775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11943.923841] Lustre: Skipped 1 previous similar message [11945.528224] Lustre: server umount lustre-MDT0000 complete [11949.034180] LustreError: 397296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11949.061470] LustreError: 397296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11949.422136] LustreError: 396339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746305 with bad export cookie 16649140035340260457 [11949.428214] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11949.447511] LustreError: 396339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11949.874694] Lustre: server umount lustre-MDT0001 complete [11961.837091] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [11964.833836] Key type lgssc unregistered [11965.120279] LNet: 400725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11965.124797] LNetError: 400725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11965.148524] LNet: Removed LNI 192.168.201.135@tcp [11966.084657] Key type .llcrypt unregistered [11966.087126] Key type ._llcrypt unregistered [11983.605949] Key type ._llcrypt registered [11983.610524] Key type .llcrypt registered [11983.732477] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [12000.468953] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [12001.627766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12001.767634] alg: No test for adler32 (adler32-zlib) [12003.176828] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [12003.558588] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [12005.311437] Key type lgssc registered [12006.485787] Lustre: Echo OBD driver; http://www.lustre.org/ [12043.662948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12043.686949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.062876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12045.131321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12045.229944] Lustre: lustre-MDT0000: new disk, initializing [12045.329980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.348430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12050.582092] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12062.529699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12062.675620] Lustre: 403899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12062.754581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12062.764113] Lustre: Skipped 1 previous similar message [12062.853164] Lustre: lustre-MDT0001: new disk, initializing [12062.959520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12063.012462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12063.035513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12067.618223] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12078.784825] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12084.689514] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12091.955722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12092.261275] Lustre: lustre-OST0000: new disk, initializing [12092.264848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12092.272132] Lustre: 405864:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12092.368822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12098.134692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12098.141836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12098.230686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12098.917199] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12109.125443] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12114.377260] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12114.597987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12119.335562] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12119.539913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12124.129995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12124.134698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12124.145794] LustreError: Skipped 1 previous similar message [12124.154235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12124.204417] Lustre: Skipped 1 previous similar message [12128.022787] Lustre: server umount lustre-OST0000 complete [12134.372204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12134.381264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12134.397598] Lustre: Skipped 1 previous similar message [12134.414938] Lustre: Skipped 2 previous similar messages [12138.267416] Lustre: server umount lustre-MDT0000 complete [12142.496356] LustreError: 403892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746498 with bad export cookie 1435726850737522709 [12142.517571] LustreError: 403892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12142.519343] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12142.834359] Lustre: server umount lustre-MDT0001 complete [12153.168734] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12155.067220] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:35:09 (1790746509) [12163.132711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12163.605889] LustreError: 408327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12163.630240] LustreError: 408327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12163.719140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12169.187370] LustreError: 408328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.358410] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12173.289105] LustreError: 408327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12178.402847] LustreError: 408328:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12178.591491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12178.984747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12184.428384] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12192.479672] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12198.803327] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12206.558412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12206.947633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12214.653986] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12223.166829] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12225.370126] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12228.146563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12234.727309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12234.727941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12234.752289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12234.759571] Lustre: Skipped 1 previous similar message [12239.842600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12239.855811] Lustre: Skipped 2 previous similar messages [12240.748796] Lustre: server umount lustre-OST0000 complete [12245.473092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12245.485731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.495897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12250.087607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12253.452791] Lustre: server umount lustre-MDT0000 complete [12255.210573] LustreError: 408322:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.239630] LustreError: 408322:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12257.824511] LustreError: 409062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746613 with bad export cookie 1435726850737523598 [12257.827290] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12257.833269] LustreError: 409062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12258.345427] Lustre: server umount lustre-MDT0001 complete [12270.026974] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [12273.287949] Key type lgssc unregistered [12273.539716] LNet: 412056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12273.546958] LNetError: 412056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12273.572718] LNet: Removed LNI 192.168.201.135@tcp [12274.487173] Key type .llcrypt unregistered [12274.489446] Key type ._llcrypt unregistered [12303.354980] Key type ._llcrypt registered [12303.358597] Key type .llcrypt registered [12303.490620] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [12303.821366] Key type .llcrypt unregistered [12303.823485] Key type ._llcrypt unregistered [12325.017630] Key type ._llcrypt registered [12325.019407] Key type .llcrypt registered [12325.185593] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [12326.493191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12326.668308] alg: No test for adler32 (adler32-zlib) [12327.749963] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [12328.019884] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [12329.736116] Key type lgssc registered [12331.368784] Lustre: Echo OBD driver; http://www.lustre.org/ [12339.777095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12339.820563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12341.331519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12346.053066] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12351.466075] LustreError: 414284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.488562] LustreError: 414284:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12355.714742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12356.044977] LustreError: 414285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12356.077486] LustreError: 414285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12356.099893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12361.185246] LustreError: 414284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12361.201751] LustreError: 414284:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12362.607817] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12366.316824] LustreError: 414285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.348928] LustreError: 414285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12366.360800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12370.279348] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12371.431888] LustreError: 414284:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12371.454518] LustreError: 414284:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12376.638191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12377.083243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12381.781247] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12391.905080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12391.917412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12391.946229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12392.377444] Lustre: server umount lustre-OST0000 complete [12397.434827] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12397.442157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12397.451513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12398.041848] Lustre: server umount lustre-MDT0000 complete [12402.680838] LustreError: 414281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12402.720973] LustreError: 414281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12402.733014] LustreError: 414265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746758 with bad export cookie 15691541860897608157 [12402.733941] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12402.752868] LustreError: 414265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12413.030514] Lustre: server umount lustre-MDT0001 complete [12424.231614] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [12427.144980] Key type lgssc unregistered [12427.538910] LNet: 417431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12427.550453] LNetError: 417431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12427.580196] LNet: Removed LNI 192.168.201.135@tcp [12428.822662] Key type .llcrypt unregistered [12428.825349] Key type ._llcrypt unregistered [12453.694715] Key type ._llcrypt registered [12453.696972] Key type .llcrypt registered [12453.834917] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [12455.133987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12455.227135] alg: No test for adler32 (adler32-zlib) [12456.306258] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [12456.559621] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [12458.215830] Key type lgssc registered [12459.442593] Lustre: Echo OBD driver; http://www.lustre.org/ [12466.603186] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12466.642629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12482.207772] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12487.328478] LustreError: 418835:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.135@tcp: failed processing log, type 4: rc = -110 [12514.020149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12521.599808] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12529.256649] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12536.599982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12537.220374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12537.321994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12542.437498] LustreError: 418862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12542.478291] LustreError: 418862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12543.077176] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12547.578904] LustreError: 418861:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.583775] LustreError: 418862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.599446] LustreError: 418862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12552.422043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12552.950800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12558.245992] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12566.602795] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12572.962693] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12582.862394] Lustre: server umount lustre-OST0000 complete [12586.945779] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12586.961975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12586.986980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12589.027662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12589.044766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12594.146904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12594.154499] Lustre: Skipped 2 previous similar messages [12596.319263] Lustre: server umount lustre-MDT0000 complete [12599.263187] Lustre: 418165:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790746939/real 1790746939] req@00000000fae7c0aa x1877734129170560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790746955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12599.267551] LustreError: 419875:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.267730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12599.279105] Lustre: 418165:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12599.302988] LustreError: 419875:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12600.430868] LustreError: 418842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790746956 with bad export cookie 8283695473808010190 [12600.434097] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12600.446665] LustreError: 418842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12600.874827] Lustre: server umount lustre-MDT0001 complete [12611.679797] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [12614.422799] Key type lgssc unregistered [12614.646539] LNet: 422403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12614.653386] LNetError: 422403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12614.674627] LNet: Removed LNI 192.168.201.135@tcp [12615.532969] Key type .llcrypt unregistered [12615.535210] Key type ._llcrypt unregistered [12641.584397] Key type ._llcrypt registered [12641.585922] Key type .llcrypt registered [12641.705859] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [12642.804629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12643.205606] alg: No test for adler32 (adler32-zlib) [12644.350393] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [12644.606939] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [12646.287289] Key type lgssc registered [12648.081475] Lustre: Echo OBD driver; http://www.lustre.org/ [12656.151305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12656.176788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12657.720181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12663.534773] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12667.881816] LustreError: 423836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12667.908826] LustreError: 423836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12668.904030] LustreError: 423837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12672.592447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12672.842405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12678.041474] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12686.971938] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12693.796741] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12701.681634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12702.101344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12704.122449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12707.404133] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12715.599500] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12722.497143] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.35@tcp (stopping) [12722.659831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12722.697024] Lustre: Skipped 1 previous similar message [12725.941809] Lustre: server umount lustre-MDT0000 complete [12727.548675] LustreError: 423832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12728.547696] LustreError: 423818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.35@tcp arrived at 1790747084 with bad export cookie 7502725762749343420 [12728.575051] LustreError: 423818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12730.524294] LustreError: 423819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747086 with bad export cookie 7502725762749342531 [12730.534776] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12732.897555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12732.907453] LustreError: 425023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12732.909130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12732.909138] Lustre: Skipped 1 previous similar message [12732.911332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12732.911339] Lustre: Skipped 3 previous similar messages [12732.948737] LustreError: 425023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12735.909086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12736.727734] Lustre: server umount lustre-OST0000 complete [12741.039525] LustreError: 425135:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12741.063699] LustreError: 425135:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12743.933863] Lustre: server umount lustre-MDT0001 complete [12757.037656] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [12760.234704] Key type lgssc unregistered [12760.593143] LNet: 427524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12760.605953] LNetError: 427524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12760.626057] LNet: Removed LNI 192.168.201.135@tcp [12761.776890] Key type .llcrypt unregistered [12761.779808] Key type ._llcrypt unregistered [12789.839529] Key type ._llcrypt registered [12789.841279] Key type .llcrypt registered [12789.990486] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [12791.328304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12791.466431] alg: No test for adler32 (adler32-zlib) [12792.561492] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [12792.832456] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [12794.551309] Key type lgssc registered [12796.031357] Lustre: Echo OBD driver; http://www.lustre.org/ [12804.894956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12804.909292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12806.412900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12811.608168] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12816.868704] LustreError: 428958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12816.912565] LustreError: 428958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12821.056383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12821.364669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12826.130879] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12834.104742] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12839.124104] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12846.681869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12847.025659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12852.158451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12856.494907] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [12866.597764] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12872.680765] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12872.691780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12872.696441] Lustre: Skipped 1 previous similar message [12872.716124] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12872.716311] Lustre: Skipped 1 previous similar message [12874.145592] Lustre: server umount lustre-MDT0001 complete [12874.510464] LustreError: 428954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12877.795545] LustreError: 431319:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12877.832905] LustreError: 431319:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12882.913118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12882.917813] LustreError: 429686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12882.931522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12883.007084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12885.770797] Lustre: server umount lustre-OST0000 complete [12888.039205] LustreError: 428959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12890.521304] Lustre: server umount lustre-MDT0000 complete [12906.331085] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [12910.070837] Key type lgssc unregistered [12910.459675] LNet: 432644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12910.476048] LNetError: 432644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12910.511092] LNet: Removed LNI 192.168.201.135@tcp [12911.993256] Key type .llcrypt unregistered [12911.999404] Key type ._llcrypt unregistered [12941.746463] Key type ._llcrypt registered [12941.748336] Key type .llcrypt registered [12941.850461] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [12960.556797] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [12961.659945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12961.682110] alg: No test for adler32 (adler32-zlib) [12962.775111] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [12963.056275] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [12964.727193] Key type lgssc registered [12965.873679] Lustre: Echo OBD driver; http://www.lustre.org/ [13009.594406] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [13026.537806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13026.578250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13027.823274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13027.848876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13027.955570] Lustre: lustre-MDT0000: new disk, initializing [13028.128898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13028.164502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13033.078762] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13048.646780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13049.001581] Lustre: lustre-OST0000: new disk, initializing [13049.013606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13049.022613] Lustre: Skipped 1 previous similar message [13049.025806] Lustre: 437697:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13049.116847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13057.255951] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13057.771669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13057.796284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13057.848328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13069.041129] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13085.979921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.194204] Lustre: 436676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13086.307788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13086.405608] Lustre: lustre-MDT0001: new disk, initializing [13086.611768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13086.667919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13086.686300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13091.394062] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13101.934618] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13102.113088] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13107.095919] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13107.281808] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13112.611115] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.35@tcp (stopping) [13114.867966] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13114.875737] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13114.903838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13117.697746] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.35@tcp (stopping) [13117.707816] Lustre: Skipped 1 previous similar message [13119.148819] Lustre: server umount lustre-MDT0001 complete [13125.092628] LustreError: 436690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13125.109489] LustreError: 436690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13130.213524] LustreError: 439393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13130.236055] LustreError: 439393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13135.341175] LustreError: 436690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13135.374228] LustreError: 436690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13139.936706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13139.956767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13139.971907] Lustre: Skipped 1 previous similar message [13139.981498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13140.468679] LustreError: 439394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13144.046513] Lustre: server umount lustre-OST0000 complete [13145.568487] LustreError: 436690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13155.771948] Lustre: server umount lustre-MDT0000 complete [13171.028770] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [13173.982463] Key type lgssc unregistered [13174.325253] LNet: 441445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13174.341430] LNetError: 441445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13175.402646] LNet: Removed LNI 192.168.201.135@tcp [13176.580566] Key type .llcrypt unregistered [13176.582751] Key type ._llcrypt unregistered [13222.751591] Key type ._llcrypt registered [13222.755521] Key type .llcrypt registered [13222.907866] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [13223.467335] Key type .llcrypt unregistered [13223.473809] Key type ._llcrypt unregistered [13242.079653] Key type ._llcrypt registered [13242.085549] Key type .llcrypt registered [13242.231158] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [13243.783273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13243.967629] alg: No test for adler32 (adler32-zlib) [13245.277919] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [13245.476663] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [13247.175700] Key type lgssc registered [13248.613835] Lustre: Echo OBD driver; http://www.lustre.org/ [13277.675967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13277.709094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13279.016577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13279.066067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13279.216339] Lustre: lustre-MDT0000: new disk, initializing [13279.385403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13279.421164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13284.571869] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13297.440198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13297.559397] Lustre: 444841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13297.654727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13297.665285] Lustre: Skipped 1 previous similar message [13297.790760] Lustre: lustre-MDT0001: new disk, initializing [13297.902299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13297.954598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13297.965869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13303.152117] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13313.257434] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13319.639833] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13327.267514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13327.572708] Lustre: lustre-OST0000: new disk, initializing [13327.575983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13327.581319] Lustre: 446809:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13327.656720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13334.623890] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13336.688310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13336.702726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13336.805905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13344.288934] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13352.424583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13352.429402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13352.440214] Lustre: Skipped 1 previous similar message [13352.459379] Lustre: Skipped 2 previous similar messages [13356.823691] Lustre: server umount lustre-MDT0000 complete [13360.686239] LustreError: 444833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747716 with bad export cookie 13740439061157246826 [13360.701265] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13360.703883] LustreError: 444833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13361.087171] Lustre: server umount lustre-MDT0001 complete [13376.050123] Lustre: server umount lustre-OST0000 complete [13386.801547] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:55:41 (1790747741) [13395.528617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13396.065185] LustreError: 448837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13396.078307] LustreError: 448837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13396.149478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13401.253296] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13401.573172] LustreError: 448838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13405.680866] LustreError: 448837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13410.060112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13410.424564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13415.440625] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13423.293799] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13429.373062] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13449.909809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13450.185914] Lustre: Found index 0 for lustre-OST0000, updating log [13450.487114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13456.825394] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [13466.022174] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13466.935991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13477.358515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13477.367489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13477.369322] Lustre: Skipped 1 previous similar message [13477.370062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13478.442924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13478.463573] Lustre: Skipped 2 previous similar messages [13482.886426] Lustre: server umount lustre-OST0000 complete [13502.432694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13502.445796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13502.461938] Lustre: Skipped 1 previous similar message [13502.485222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13504.546454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13506.715742] Lustre: server umount lustre-MDT0000 complete [13509.663976] LNet: 1 local NIs in recovery (showing 1): 192.168.201.135@tcp [13511.146865] LustreError: 448833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13511.174170] LustreError: 448833:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13511.240884] LustreError: 448819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790747867 with bad export cookie 13740439061157248219 [13511.262936] LustreError: 448819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13511.266690] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13511.963179] Lustre: server umount lustre-MDT0001 complete [13521.986837] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:57:56 (1790747876) [14147.912970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14148.549160] LustreError: 455266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14148.605166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14153.587976] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14153.705424] LustreError: 455267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14158.826365] LustreError: 455266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14162.174164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14162.842759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14169.761228] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14178.487729] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14184.107319] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14200.867699] LNet: 1 local NIs in recovery (showing 1): 192.168.201.135@tcp [14205.185652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14205.408627] Lustre: Found index 0 for lustre-OST0000, updating log [14205.994799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14214.925333] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14224.943505] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14228.011552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14228.024858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14228.033373] Lustre: Skipped 4 previous similar messages [14241.247202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14241.555275] Lustre: server umount lustre-OST0000 complete [14246.440490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14246.450876] Lustre: Skipped 1 previous similar message [14246.483044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14247.522825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14247.547256] Lustre: Skipped 2 previous similar messages [14249.569148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14249.576703] Lustre: Skipped 3 previous similar messages [14253.051752] Lustre: server umount lustre-MDT0000 complete [14253.731757] LustreError: 455262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14253.758188] LustreError: 455262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14257.690655] LustreError: 455247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748613 with bad export cookie 13740439061157249304 [14257.714269] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14258.415946] Lustre: server umount lustre-MDT0001 complete [14260.255228] LNet: 1 local NIs in recovery (showing 1): 192.168.201.135@tcp [14292.769247] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [14305.443712] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [14348.796764] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:11:43 (1790748703) [14368.434525] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [14371.629662] Key type lgssc unregistered [14372.013960] LNet: 461883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14372.024952] LNetError: 461883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14468.329860] LNet: Removed LNI 192.168.201.135@tcp [14469.193737] Key type .llcrypt unregistered [14469.196710] Key type ._llcrypt unregistered [14489.142251] Key type ._llcrypt registered [14489.145340] Key type .llcrypt registered [14489.251370] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [14491.014906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14491.126347] alg: No test for adler32 (adler32-zlib) [14492.260303] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [14492.528468] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [14494.191252] Key type lgssc registered [14495.427712] Lustre: Echo OBD driver; http://www.lustre.org/ [14500.439361] LNet: Added LNI 192.168.201.135@tcp100 [8/256/0/180] [14501.385254] LNet: Added LNI 192.168.201.135@tcp101 [8/256/0/180] [14502.357527] LNet: Added LNI 192.168.201.135@tcp102 [8/256/0/180] [14503.352585] LNet: Added LNI 192.168.201.135@tcp103 [8/256/0/180] [14504.372445] LNet: Added LNI 192.168.201.135@tcp104 [8/256/0/180] [14505.489403] LNet: Added LNI 192.168.201.135@tcp105 [8/256/0/180] [14506.566607] LNet: Added LNI 192.168.201.135@tcp106 [8/256/0/180] [14507.628564] LNet: Added LNI 192.168.201.135@tcp107 [8/256/0/180] [14508.654765] LNet: Added LNI 192.168.201.135@tcp108 [8/256/0/180] [14509.668430] LNet: Added LNI 192.168.201.135@tcp109 [8/256/0/180] [14510.519786] LNet: Added LNI 192.168.201.135@tcp110 [8/256/0/180] [14511.530723] LNet: Added LNI 192.168.201.135@tcp111 [8/256/0/180] [14512.522920] LNet: Added LNI 192.168.201.135@tcp112 [8/256/0/180] [14513.421961] LNet: Added LNI 192.168.201.135@tcp113 [8/256/0/180] [14514.150479] LNet: Added LNI 192.168.201.135@tcp114 [8/256/0/180] [14515.214181] LNet: Added LNI 192.168.201.135@tcp115 [8/256/0/180] [14516.149555] LNet: Added LNI 192.168.201.135@tcp116 [8/256/0/180] [14517.172819] LNet: Added LNI 192.168.201.135@tcp117 [8/256/0/180] [14518.182667] LNet: Added LNI 192.168.201.135@tcp118 [8/256/0/180] [14519.085762] LNet: Added LNI 192.168.201.135@tcp119 [8/256/0/180] [14520.030841] LNet: Added LNI 192.168.201.135@tcp120 [8/256/0/180] [14521.086740] LNet: Added LNI 192.168.201.135@tcp121 [8/256/0/180] [14522.175366] LNet: Added LNI 192.168.201.135@tcp122 [8/256/0/180] [14523.085567] LNet: Added LNI 192.168.201.135@tcp123 [8/256/0/180] [14523.994645] LNet: Added LNI 192.168.201.135@tcp124 [8/256/0/180] [14524.935612] LNet: Added LNI 192.168.201.135@tcp125 [8/256/0/180] [14525.833021] LNet: Added LNI 192.168.201.135@tcp126 [8/256/0/180] [14526.859379] LNet: Added LNI 192.168.201.135@tcp127 [8/256/0/180] [14527.802616] LNet: Added LNI 192.168.201.135@tcp128 [8/256/0/180] [14528.692702] LNet: Added LNI 192.168.201.135@tcp129 [8/256/0/180] [14529.668580] LNet: Added LNI 192.168.201.135@tcp130 [8/256/0/180] [14530.480762] LNet: Added LNI 192.168.201.135@tcp131 [8/256/0/180] [14531.314741] LNet: Added LNI 192.168.201.135@tcp132 [8/256/0/180] [14532.232478] LNet: Added LNI 192.168.201.135@tcp133 [8/256/0/180] [14532.974979] LNet: Added LNI 192.168.201.135@tcp134 [8/256/0/180] [14533.894607] LNet: Added LNI 192.168.201.135@tcp135 [8/256/0/180] [14540.947699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [14541.000982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14542.185409] Lustre: 464929:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.135@tcp: can't add NID 192.168.201.135@tcp130: rc = -75 [14542.286204] LNetError: 462375:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.135 [14542.307396] LNetError: Protocol error connecting to 192.168.201.135@tcp119 at host 192.168.201.135:988: is it running a compatible version of Lustre? [14542.456340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14542.519368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14542.591883] Lustre: lustre-MDT0000: new disk, initializing [14542.663740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14542.692276] LNet: 6 local NIs in recovery (showing 6): 192.168.201.135@tcp123, 192.168.201.135@tcp125, 192.168.201.135@tcp126, 192.168.201.135@tcp127, 192.168.201.135@tcp128, 192.168.201.135@tcp124 [14542.698236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14543.399057] LNetError: 464950:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.135 [14543.411952] LNetError: 464950:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14543.430191] LNetError: Protocol error connecting to 192.168.201.135@tcp103 at host 192.168.201.135:988: is it running a compatible version of Lustre? [14543.443040] LNetError: Skipped 11 previous similar messages [14545.762186] LNetError: 462374:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.135 [14545.776814] LNetError: 462374:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14545.792939] LNetError: Protocol error connecting to 192.168.201.135@tcp125 at host 192.168.201.135:988: is it running a compatible version of Lustre? [14545.825608] LNetError: Skipped 10 previous similar messages [14548.826087] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14549.856874] LNetError: 464958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.135 [14549.868772] LNetError: 464958:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14549.879343] LNetError: Protocol error connecting to 192.168.201.135@tcp125 at host 192.168.201.135:988: is it running a compatible version of Lustre? [14549.889837] LNetError: Skipped 1 previous similar message [14557.025370] LNetError: 464962:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.135 [14557.034810] LNetError: Protocol error connecting to 192.168.201.135@tcp125 at host 192.168.201.135:988: is it running a compatible version of Lustre? [14561.481557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14561.765386] Lustre: 464946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14561.927791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14561.933712] Lustre: Skipped 1 previous similar message [14562.041917] Lustre: lustre-MDT0001: new disk, initializing [14562.094991] Lustre: 465749:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.135@tcp: can't add NID 192.168.201.135@tcp130: rc = -75 [14562.100198] Lustre: 465749:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14562.246506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14562.290288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14562.303802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14567.389318] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14569.507419] Lustre: 466112:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.135@tcp: can't add NID 192.168.201.135@tcp130: rc = -75 [14569.521781] Lustre: 466112:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14572.411672] LNetError: 462376:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.135 [14572.426709] LNetError: Protocol error connecting to 192.168.201.135@tcp122 at host 192.168.201.135:988: is it running a compatible version of Lustre? [14577.736568] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14583.931851] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14588.776136] LNetError: 464955:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.135 [14588.791709] LNetError: 464955:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [14588.797159] LNetError: Protocol error connecting to 192.168.201.135@tcp120 at host 192.168.201.135:988: is it running a compatible version of Lustre? [14588.812754] LNetError: Skipped 13 previous similar messages [14591.827914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14592.333841] Lustre: lustre-OST0000: new disk, initializing [14592.350550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14592.368154] Lustre: 466921:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14592.516958] Lustre: 466926:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.135@tcp: can't add NID 192.168.201.135@tcp130: rc = -75 [14592.525584] Lustre: 466926:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14592.579932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14600.339716] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14600.740141] Lustre: 467295:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.135@tcp: can't add NID 192.168.201.135@tcp130: rc = -75 [14600.751885] Lustre: 467295:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14600.820924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14600.841311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14601.026717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14603.103163] LNet: 1 local NIs in recovery (showing 1): 192.168.201.135@tcp125 [14610.228625] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14621.540395] LNetError: 462377:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.135 [14621.549959] LNetError: 462377:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 27 previous similar messages [14621.566727] LNetError: Protocol error connecting to 192.168.201.135@tcp119 at host 192.168.201.135:988: is it running a compatible version of Lustre? [14621.578513] LNetError: Skipped 27 previous similar messages [14621.664172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14621.669842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14621.673368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14621.689583] Lustre: Skipped 1 previous similar message [14624.511557] Lustre: server umount lustre-OST0000 complete [14628.833759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14628.841175] LustreError: Skipped 1 previous similar message [14628.847043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14628.864144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14628.868750] Lustre: Skipped 1 previous similar message [14631.904653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14631.921837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14634.619814] Lustre: server umount lustre-MDT0000 complete [14638.240804] LustreError: 464937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790748994 with bad export cookie 4388915634970813711 [14638.247752] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14638.250574] LustreError: 464937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14638.575931] Lustre: server umount lustre-MDT0001 complete [14651.014353] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [14653.553352] Key type lgssc unregistered [14653.990804] LNet: 468834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14654.004153] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14654.011635] LNet: Removed LNI 192.168.201.135@tcp [14655.071329] LNet: Removed LNI 192.168.201.135@tcp100 [14655.077695] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14655.099772] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14655.110517] LNet: Removed LNI 192.168.201.135@tcp101 [14656.159323] LNet: Removed LNI 192.168.201.135@tcp102 [14656.170346] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14656.182829] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14656.194746] LNet: Removed LNI 192.168.201.135@tcp103 [14657.247150] LNet: Removed LNI 192.168.201.135@tcp104 [14658.271193] LNet: Removed LNI 192.168.201.135@tcp105 [14658.273588] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14658.286486] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14658.300876] LNet: Removed LNI 192.168.201.135@tcp106 [14659.359480] LNet: Removed LNI 192.168.201.135@tcp107 [14660.447165] LNet: Removed LNI 192.168.201.135@tcp108 [14661.471196] LNet: Removed LNI 192.168.201.135@tcp109 [14662.495421] LNet: Removed LNI 192.168.201.135@tcp110 [14662.500804] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14662.506356] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14663.519202] LNet: Removed LNI 192.168.201.135@tcp111 [14664.543204] LNet: Removed LNI 192.168.201.135@tcp112 [14665.567190] LNet: Removed LNI 192.168.201.135@tcp113 [14666.592106] LNet: Removed LNI 192.168.201.135@tcp114 [14667.615577] LNet: Removed LNI 192.168.201.135@tcp115 [14668.639183] LNet: Removed LNI 192.168.201.135@tcp116 [14669.663147] LNet: Removed LNI 192.168.201.135@tcp117 [14670.687308] LNet: Removed LNI 192.168.201.135@tcp118 [14670.699700] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14670.716536] LNetError: 468834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [14670.727234] LNet: Removed LNI 192.168.201.135@tcp119 [14671.775233] LNet: Removed LNI 192.168.201.135@tcp120 [14672.799391] LNet: Removed LNI 192.168.201.135@tcp121 [14673.823193] LNet: Removed LNI 192.168.201.135@tcp122 [14674.847311] LNet: Removed LNI 192.168.201.135@tcp123 [14675.871202] LNet: Removed LNI 192.168.201.135@tcp124 [14675.874424] LNet: Removed LNI 192.168.201.135@tcp125 [14676.895276] LNet: Removed LNI 192.168.201.135@tcp126 [14677.919284] LNet: Removed LNI 192.168.201.135@tcp127 [14678.943468] LNet: Removed LNI 192.168.201.135@tcp128 [14679.967164] LNet: Removed LNI 192.168.201.135@tcp129 [14679.970333] LNet: Removed LNI 192.168.201.135@tcp130 [14679.974668] LNet: Removed LNI 192.168.201.135@tcp131 [14679.981592] LNet: Removed LNI 192.168.201.135@tcp132 [14679.985101] LNet: Removed LNI 192.168.201.135@tcp133 [14679.990863] LNet: Removed LNI 192.168.201.135@tcp134 [14680.007954] LNet: Removed LNI 192.168.201.135@tcp135 [14680.753287] Key type .llcrypt unregistered [14680.754684] Key type ._llcrypt unregistered [14703.433509] Key type ._llcrypt registered [14703.435757] Key type .llcrypt registered [14703.645087] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [14720.690280] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [14722.327772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14722.390969] alg: No test for adler32 (adler32-zlib) [14723.468726] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [14723.784954] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [14725.479210] Key type lgssc registered [14726.585858] Lustre: Echo OBD driver; http://www.lustre.org/ [14763.454823] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:18:38 (1790749118) [14780.661090] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [14783.027827] Key type lgssc unregistered [14783.279598] LNet: 472937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14783.282244] LNetError: 472937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14783.294817] LNet: Removed LNI 192.168.201.135@tcp [14784.030763] Key type .llcrypt unregistered [14784.036071] Key type ._llcrypt unregistered [14799.828368] Key type ._llcrypt registered [14799.831952] Key type .llcrypt registered [14799.965638] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [14800.958261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14801.029103] alg: No test for adler32 (adler32-zlib) [14802.154884] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [14802.412478] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [14804.071306] Key type lgssc registered [14805.287581] Lustre: Echo OBD driver; http://www.lustre.org/ [14814.028630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [14814.057444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14815.273491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14815.296775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14815.354517] Lustre: lustre-MDT0000: new disk, initializing [14815.416264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14815.445084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14819.986025] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14829.370473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14829.517326] Lustre: 474167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14829.579848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14829.583156] Lustre: Skipped 1 previous similar message [14829.696516] Lustre: lustre-MDT0001: new disk, initializing [14829.792070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14829.825403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14829.841372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14834.141592] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14842.412935] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14847.887713] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14854.713203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14854.975477] Lustre: lustre-OST0000: new disk, initializing [14854.983430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14854.989864] Lustre: 476137:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14855.074538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14861.851850] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [14862.399701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14862.405786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14862.561856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14870.913815] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14874.344521] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing lnet_if_list [14879.392211] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14879.418542] LNet: Added LNI 192.168.201.135@tcp10 [8/256/0/180] [14885.519969] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14885.540565] LNet: Added LNI 192.168.201.135@tcp11 [8/256/0/180] [14889.957541] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14889.977956] LNet: Added LNI 192.168.201.135@tcp12 [8/256/0/180] [14894.204495] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14894.232037] LNet: Added LNI 192.168.201.135@tcp13 [8/256/0/180] [14899.308511] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14899.361792] LNet: Added LNI 192.168.201.135@tcp14 [8/256/0/180] [14904.944547] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14904.989243] LNet: Added LNI 192.168.201.135@tcp15 [8/256/0/180] [14910.501778] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14910.554677] LNet: Added LNI 192.168.201.135@tcp16 [8/256/0/180] [14915.633703] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14915.680166] LNet: Added LNI 192.168.201.135@tcp17 [8/256/0/180] [14920.893499] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14920.929307] LNet: Added LNI 192.168.201.135@tcp18 [8/256/0/180] [14926.799236] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14926.826282] LNet: Added LNI 192.168.201.135@tcp19 [8/256/0/180] [14932.301865] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14932.339398] LNet: Added LNI 192.168.201.135@tcp20 [8/256/0/180] [14937.842885] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14937.867148] LNet: Added LNI 192.168.201.135@tcp21 [8/256/0/180] [14942.801988] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14942.823575] LNet: Added LNI 192.168.201.135@tcp22 [8/256/0/180] [14949.597965] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14949.627931] LNet: Added LNI 192.168.201.135@tcp23 [8/256/0/180] [14955.172400] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14955.189296] LNet: Added LNI 192.168.201.135@tcp24 [8/256/0/180] [14961.143233] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14961.166590] LNet: Added LNI 192.168.201.135@tcp25 [8/256/0/180] [14968.136356] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14968.187786] LNet: Added LNI 192.168.201.135@tcp26 [8/256/0/180] [14974.786904] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14974.802686] LNet: Added LNI 192.168.201.135@tcp27 [8/256/0/180] [14981.220853] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14981.247323] LNet: Added LNI 192.168.201.135@tcp28 [8/256/0/180] [14987.975781] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14988.018615] LNet: Added LNI 192.168.201.135@tcp29 [8/256/0/180] [14994.430333] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14994.467054] LNet: Added LNI 192.168.201.135@tcp30 [8/256/0/180] [15001.841423] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15001.888203] LNet: Added LNI 192.168.201.135@tcp31 [8/256/0/180] [15008.683665] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15008.704665] LNet: Added LNI 192.168.201.135@tcp32 [8/256/0/180] [15015.454677] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15015.487306] LNet: Added LNI 192.168.201.135@tcp33 [8/256/0/180] [15022.652780] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15022.675521] LNet: Added LNI 192.168.201.135@tcp34 [8/256/0/180] [15030.011320] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15030.044819] LNet: Added LNI 192.168.201.135@tcp35 [8/256/0/180] [15038.937437] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15038.991933] LNet: Added LNI 192.168.201.135@tcp36 [8/256/0/180] [15046.287570] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15046.332787] LNet: Added LNI 192.168.201.135@tcp37 [8/256/0/180] [15052.998913] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15053.031665] LNet: Added LNI 192.168.201.135@tcp38 [8/256/0/180] [15059.471115] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15059.517330] LNet: Added LNI 192.168.201.135@tcp39 [8/256/0/180] [15066.213868] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15066.247908] LNet: Added LNI 192.168.201.135@tcp40 [8/256/0/180] [15073.614329] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15073.644498] LNet: Added LNI 192.168.201.135@tcp41 [8/256/0/180] [15082.032919] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15082.090609] LNet: Added LNI 192.168.201.135@tcp42 [8/256/0/180] [15091.002481] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15091.035834] LNet: Added LNI 192.168.201.135@tcp43 [8/256/0/180] [15098.817277] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15098.857098] LNet: Added LNI 192.168.201.135@tcp44 [8/256/0/180] [15107.372682] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15107.440951] LNet: Added LNI 192.168.201.135@tcp45 [8/256/0/180] [15115.327883] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15115.348796] LNet: Added LNI 192.168.201.135@tcp46 [8/256/0/180] [15123.964451] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15124.002400] LNet: Added LNI 192.168.201.135@tcp47 [8/256/0/180] [15133.367812] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15133.400996] LNet: Added LNI 192.168.201.135@tcp48 [8/256/0/180] [15142.856823] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15142.893565] LNet: Added LNI 192.168.201.135@tcp49 [8/256/0/180] [15150.350166] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15150.379461] LNet: Added LNI 192.168.201.135@tcp50 [8/256/0/180] [15167.969161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15167.972559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15167.978087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15170.027839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15170.030186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15170.044948] Lustre: Skipped 1 previous similar message [15170.053043] Lustre: Skipped 2 previous similar messages [15172.600798] Lustre: server umount lustre-MDT0000 complete [15175.822633] LustreError: 474159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749531 with bad export cookie 13431997855066546071 [15175.826421] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15175.844483] LustreError: 474159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15176.502528] Lustre: server umount lustre-MDT0001 complete [15196.639255] Lustre: 473431:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790749536/real 1790749536] req@00000000c0413518 x1877736589276032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790749552 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15196.666492] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15197.395310] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [15201.759226] Lustre: 473429:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790749541/real 1790749541] req@000000003c1410e9 x1877736589276288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790749557 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15206.948484] Lustre: 473429:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790749546/real 1790749546] req@0000000022987e60 x1877736589276416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790749562 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15208.132889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15212.001024] Lustre: 473430:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790749551/real 1790749551] req@000000000835a99c x1877736589276544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790749567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15220.255852] LustreError: 473427:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008a8d9ae7 x1877736589277184/t0(0) o250->MGC192.168.201.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15220.687563] LustreError: 476130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15220.722623] LustreError: 476130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15220.793425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15221.794769] LustreError: 476131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15221.823215] LustreError: 476131:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15225.891067] LustreError: 500380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15225.893881] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15225.908062] LustreError: 500380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15226.384387] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15230.949517] LustreError: 476131:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15230.970262] LustreError: 476131:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15235.983810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15236.065796] LustreError: 500379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15236.098148] LustreError: 500379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15236.437215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15241.718718] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15242.006289] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15250.318989] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15256.713365] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15277.028128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15277.046350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15277.079492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15282.696297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15282.708096] Lustre: Skipped 2 previous similar messages [15287.787916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15287.809488] Lustre: Skipped 1 previous similar message [15290.335816] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15290.493990] Lustre: server umount lustre-OST0000 complete [15298.020973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15298.040022] Lustre: Skipped 2 previous similar messages [15298.049909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15298.069035] Lustre: Skipped 1 previous similar message [15301.574348] Lustre: server umount lustre-MDT0000 complete [15303.137370] LustreError: 500380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15303.144574] LustreError: 500380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15304.514899] LustreError: 474159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749660 with bad export cookie 13431997855066548780 [15304.521992] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15304.526609] LustreError: 474159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15304.765280] Lustre: server umount lustre-MDT0001 complete [15317.076495] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [15320.219868] Key type lgssc unregistered [15320.722405] LNet: 503160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15320.726753] LNetError: 503160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15321.761292] LNet: Removed LNI 192.168.201.135@tcp [15321.764333] LNetError: 503160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15321.780478] LNet: Removed LNI 192.168.201.135@tcp10 [15321.784387] LNet: Removed LNI 192.168.201.135@tcp11 [15321.791674] LNet: Removed LNI 192.168.201.135@tcp12 [15321.800313] LNet: Removed LNI 192.168.201.135@tcp13 [15321.811955] LNet: Removed LNI 192.168.201.135@tcp14 [15321.821077] LNet: Removed LNI 192.168.201.135@tcp15 [15321.829458] LNet: Removed LNI 192.168.201.135@tcp16 [15321.848401] LNet: Removed LNI 192.168.201.135@tcp17 [15321.856601] LNet: Removed LNI 192.168.201.135@tcp18 [15321.860355] LNet: Removed LNI 192.168.201.135@tcp19 [15321.865060] LNet: Removed LNI 192.168.201.135@tcp20 [15321.866755] LNet: Removed LNI 192.168.201.135@tcp21 [15321.873313] LNet: Removed LNI 192.168.201.135@tcp22 [15321.884886] LNet: Removed LNI 192.168.201.135@tcp23 [15321.887952] LNet: Removed LNI 192.168.201.135@tcp24 [15321.891265] LNet: Removed LNI 192.168.201.135@tcp25 [15321.896751] LNet: Removed LNI 192.168.201.135@tcp26 [15321.921311] LNet: Removed LNI 192.168.201.135@tcp27 [15321.923166] LNet: Removed LNI 192.168.201.135@tcp28 [15321.936347] LNet: Removed LNI 192.168.201.135@tcp29 [15321.941552] LNet: Removed LNI 192.168.201.135@tcp30 [15321.945723] LNet: Removed LNI 192.168.201.135@tcp31 [15321.960605] LNet: Removed LNI 192.168.201.135@tcp32 [15321.962819] LNet: Removed LNI 192.168.201.135@tcp33 [15321.976159] LNet: Removed LNI 192.168.201.135@tcp34 [15321.981509] LNet: Removed LNI 192.168.201.135@tcp35 [15321.986714] LNet: Removed LNI 192.168.201.135@tcp36 [15322.001374] LNet: Removed LNI 192.168.201.135@tcp37 [15322.007696] LNet: Removed LNI 192.168.201.135@tcp38 [15322.014966] LNet: Removed LNI 192.168.201.135@tcp39 [15322.021688] LNet: Removed LNI 192.168.201.135@tcp40 [15322.032241] LNet: Removed LNI 192.168.201.135@tcp41 [15322.044550] LNet: Removed LNI 192.168.201.135@tcp42 [15322.060456] LNet: Removed LNI 192.168.201.135@tcp43 [15322.067281] LNet: Removed LNI 192.168.201.135@tcp44 [15322.072361] LNet: Removed LNI 192.168.201.135@tcp45 [15322.076444] LNet: Removed LNI 192.168.201.135@tcp46 [15322.078262] LNet: Removed LNI 192.168.201.135@tcp47 [15322.092917] LNet: Removed LNI 192.168.201.135@tcp48 [15322.098537] LNet: Removed LNI 192.168.201.135@tcp49 [15322.111713] LNet: Removed LNI 192.168.201.135@tcp50 [15323.132182] Key type .llcrypt unregistered [15323.136758] Key type ._llcrypt unregistered [15346.134500] Key type ._llcrypt registered [15346.136414] Key type .llcrypt registered [15346.248776] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [15363.984184] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [15366.135207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15366.278129] alg: No test for adler32 (adler32-zlib) [15367.495590] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [15367.741102] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [15369.399242] Key type lgssc registered [15370.662437] Lustre: Echo OBD driver; http://www.lustre.org/ [15409.111425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15409.142303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15410.340787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15410.365030] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15410.416524] Lustre: lustre-MDT0000: new disk, initializing [15410.516518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15410.543469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15414.653488] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15426.341782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15426.465779] Lustre: 506376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15426.517583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15426.522288] Lustre: Skipped 1 previous similar message [15426.623480] Lustre: lustre-MDT0001: new disk, initializing [15426.802985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15426.855310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15426.876983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15432.798589] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15443.292319] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15449.974073] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15459.737976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15460.058267] Lustre: *** cfs_fail_loc=90f, val=0*** [15460.127370] Lustre: lustre-OST0000: new disk, initializing [15460.131541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15460.138536] Lustre: 508394:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15460.232158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15466.928450] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15467.203527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15467.217386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15467.440793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15479.225880] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15493.088717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15493.102726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15493.107364] LustreError: Skipped 1 previous similar message [15493.108163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15493.141039] Lustre: Skipped 1 previous similar message [15494.871943] Lustre: server umount lustre-OST0000 complete [15503.328789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15503.333195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15503.357495] Lustre: Skipped 1 previous similar message [15503.377315] Lustre: Skipped 2 previous similar messages [15505.463394] Lustre: server umount lustre-MDT0000 complete [15509.507680] LustreError: 506367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790749865 with bad export cookie 7891952769467785164 [15509.518485] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15509.521815] LustreError: 506367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15509.893777] Lustre: server umount lustre-MDT0001 complete [15522.918715] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [15526.389552] Key type lgssc unregistered [15526.786751] LNet: 510300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15526.798936] LNetError: 510300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15526.832246] LNet: Removed LNI 192.168.201.135@tcp [15527.962747] Key type .llcrypt unregistered [15527.970285] Key type ._llcrypt unregistered [15551.950721] Key type ._llcrypt registered [15551.954256] Key type .llcrypt registered [15552.079563] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [15553.851187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15553.875580] alg: No test for adler32 (adler32-zlib) [15554.987702] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [15555.298254] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [15557.047267] Key type lgssc registered [15558.430423] Lustre: Echo OBD driver; http://www.lustre.org/ [15568.378723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15568.415728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15570.228498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15575.041912] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15580.651944] LustreError: 511589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15580.685849] LustreError: 511589:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15584.690304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15585.071383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15590.409316] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15598.552220] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15604.913212] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15613.656450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15614.035677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15620.753943] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15629.304211] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15639.472477] loop0: detected capacity change from 0 to 204800000 [15639.725725] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15639.735784] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15639.752389] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15645.988792] loop0: detected capacity change from 0 to 204800000 [15646.219400] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15646.226699] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15646.234515] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15651.339544] loop0: detected capacity change from 0 to 204800000 [15657.762151] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15657.971462] Lustre: fs15246-OST0000: new disk, initializing [15657.976554] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15657.988283] Lustre: 515184:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15658.067939] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15664.996980] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15675.323241] loop1: detected capacity change from 0 to 204800000 [15681.815610] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15681.973412] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15682.031367] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15682.139246] Lustre: fs15246-MDT0000: new disk, initializing [15682.419287] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15682.451932] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15682.608974] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15683.621310] Lustre: 516304:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000030aeed06 x1877737378726144/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790750055 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15687.943307] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15688.760828] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15688.781819] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15688.836140] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15713.764430] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15713.812521] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15717.384749] Lustre: server umount fs15246-MDT0000 complete [15718.887701] LustreError: 511585:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15724.010479] LustreError: 511585:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15727.012874] LustreError: 511586:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15727.610526] Lustre: server umount fs15246-OST0000 complete [15742.433388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15742.446682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15742.455606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15742.472607] Lustre: Skipped 1 previous similar message [15746.680403] Lustre: server umount lustre-OST0000 complete [15752.691302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15752.716951] Lustre: Skipped 1 previous similar message [15752.717112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15752.739100] Lustre: Skipped 2 previous similar messages [15757.696309] Lustre: server umount lustre-MDT0000 complete [15757.795260] LustreError: 511590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15757.822061] LustreError: 511590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15761.451470] LustreError: 511571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750117 with bad export cookie 8861671461402569742 [15761.454808] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15761.460043] LustreError: 511571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15761.929858] Lustre: server umount lustre-MDT0001 complete [15774.541851] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [15777.484508] Key type lgssc unregistered [15777.773758] LNet: 519399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15777.788504] LNetError: 519399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15778.856072] LNet: Removed LNI 192.168.201.135@tcp [15780.008379] Key type .llcrypt unregistered [15780.010157] Key type ._llcrypt unregistered [15800.134167] Key type ._llcrypt registered [15800.136578] Key type .llcrypt registered [15800.264370] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [15801.438435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15801.468456] alg: No test for adler32 (adler32-zlib) [15802.588382] Lustre: Lustre: Build Version: 2.17.59_3_g4e55f3f [15802.908400] LNet: Added LNI 192.168.201.135@tcp [8/256/0/180] [15804.551227] Key type lgssc registered [15805.626507] Lustre: Echo OBD driver; http://www.lustre.org/ [15849.643749] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [15860.115865] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [15900.047669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15900.081084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15901.392617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15901.426963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15901.521294] Lustre: lustre-MDT0000: new disk, initializing [15901.606994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15901.634716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15905.639338] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15916.824130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15916.983340] Lustre: 523888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15917.047593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15917.053060] Lustre: Skipped 1 previous similar message [15917.124633] Lustre: lustre-MDT0001: new disk, initializing [15917.236442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15917.260925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15917.272158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15923.547377] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15935.089424] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15941.447956] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15948.961474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15949.367288] Lustre: lustre-OST0000: new disk, initializing [15949.372663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15949.388900] Lustre: 525853:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15949.587154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15955.282583] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_default_debug -1 all [15958.190387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15958.241501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15958.369687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15965.514121] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15970.419849] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15970.597633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15975.519842] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15975.710423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15978.978165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15978.989138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15978.989595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15978.991848] Lustre: Skipped 1 previous similar message [15979.011116] Lustre: Skipped 1 previous similar message [15984.112898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15989.220495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15989.232299] Lustre: Skipped 2 previous similar messages [15992.287591] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15992.560782] Lustre: server umount lustre-OST0000 complete [15999.458469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15999.460035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15999.480972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15999.497619] Lustre: Skipped 1 previous similar message [16002.618095] Lustre: server umount lustre-MDT0000 complete [16006.714653] LustreError: 523880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790750362 with bad export cookie 10462957481002394326 [16006.724329] LustreError: MGC192.168.201.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16006.744906] LustreError: 523880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16007.205977] Lustre: server umount lustre-MDT0001 complete [16035.797955] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing set_hostid [16046.531829] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing load_modules_local [16079.207817] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15778 sec ========= 02:40:33 (1790750433) [16081.045429] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:40:35 (1790750435) === [16084.903662] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:40:39 (1790750439) === [16128.210943] Lustre: DEBUG MARKER: oleg135-server.virtnet: executing unload_modules_local [16130.997307] Key type lgssc unregistered [16131.278565] LNet: 532497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16131.285932] LNetError: 532497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16132.334129] LNet: Removed LNI 192.168.201.135@tcp [16133.042611] Key type .llcrypt unregistered [16133.050820] Key type ._llcrypt unregistered