[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489834781 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002376] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007791] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010152] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.014037] *** VALIDATE selinux *** [ 0.023749] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028205] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032116] *** VALIDATE tmpfs *** [ 0.034460] *** VALIDATE proc *** [ 0.036049] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.038294] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043029] Spectre V2 : User space: Vulnerable [ 0.044011] Speculative Store Bypass: Vulnerable [ 0.047043] debug: unmapping init [mem 0xffffffffa8e59000-0xffffffffa8e60fff] [ 0.050250] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051846] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052030] ... version: 2 [ 0.052890] ... bit width: 48 [ 0.053011] ... generic registers: 4 [ 0.053846] ... value mask: 0000ffffffffffff [ 0.054010] ... max period: 00007fffffffffff [ 0.054998] ... fixed-purpose events: 3 [ 0.055010] ... event mask: 000000070000000f [ 0.056253] rcu: Hierarchical SRCU implementation. [ 0.058392] smp: Bringing up secondary CPUs ... [ 0.059702] x86: Booting SMP configuration: [ 0.060041] .... node #0, CPUs: #1 #2 #3 [ 0.066134] smp: Brought up 1 node, 4 CPUs [ 0.068019] smpboot: Max logical packages: 1 [ 0.069013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.202044] node 0 deferred pages initialised in 131ms [ 0.206205] devtmpfs: initialized [ 0.207258] x86/mm: Memory block size: 128MB [ 0.210973] gcov: version magic: 0x41383552 [ 0.214280] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.215074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.218386] pinctrl core: initialized pinctrl subsystem [ 0.220204] [ 0.220869] ************************************************************* [ 0.224022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228017] ** ** [ 0.230013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233017] ** ** [ 0.236014] ** This means that this kernel is built to expose internal ** [ 0.239014] ** IOMMU data structures, which may compromise security on ** [ 0.241013] ** your system. ** [ 0.244015] ** ** [ 0.247029] ** If you see this message and you are not debugging the ** [ 0.250019] ** kernel, report this immediately to your vendor! ** [ 0.253025] ** ** [ 0.255017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258027] ************************************************************* [ 0.261951] NET: Registered protocol family 16 [ 0.265577] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.269246] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.274091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.279133] cpuidle: using governor menu [ 0.281720] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.285449] PCI: Using configuration type 1 for base access [ 0.288137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.298122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.301080] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.307105] cryptd: max_cpu_qlen set to 1000 [ 0.310307] ACPI: Added _OSI(Module Device) [ 0.312030] ACPI: Added _OSI(Processor Device) [ 0.314023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.316025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.321326] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.330014] ACPI: Interpreter enabled [ 0.331093] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333026] ACPI: Using IOAPIC for interrupt routing [ 0.335150] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.340565] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.352384] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.354084] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.357041] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.360081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.368772] acpiphp: Slot [2] registered [ 0.371251] acpiphp: Slot [5] registered [ 0.373179] acpiphp: Slot [6] registered [ 0.376153] acpiphp: Slot [7] registered [ 0.378141] acpiphp: Slot [8] registered [ 0.380158] acpiphp: Slot [9] registered [ 0.382213] acpiphp: Slot [10] registered [ 0.384167] acpiphp: Slot [3] registered [ 0.387159] acpiphp: Slot [4] registered [ 0.388223] acpiphp: Slot [11] registered [ 0.390138] acpiphp: Slot [12] registered [ 0.392150] acpiphp: Slot [13] registered [ 0.393177] acpiphp: Slot [14] registered [ 0.395172] acpiphp: Slot [15] registered [ 0.397152] acpiphp: Slot [16] registered [ 0.399151] acpiphp: Slot [17] registered [ 0.400308] acpiphp: Slot [18] registered [ 0.401082] acpiphp: Slot [19] registered [ 0.402086] acpiphp: Slot [20] registered [ 0.403081] acpiphp: Slot [21] registered [ 0.405145] acpiphp: Slot [22] registered [ 0.406134] acpiphp: Slot [23] registered [ 0.408211] acpiphp: Slot [24] registered [ 0.409151] acpiphp: Slot [25] registered [ 0.411149] acpiphp: Slot [26] registered [ 0.413140] acpiphp: Slot [27] registered [ 0.414312] acpiphp: Slot [28] registered [ 0.415101] acpiphp: Slot [29] registered [ 0.416064] acpiphp: Slot [30] registered [ 0.417103] acpiphp: Slot [31] registered [ 0.418050] PCI host bridge to bus 0000:00 [ 0.419017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.421032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.426024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.428030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429229] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432023] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.444769] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.450086] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.452029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.454033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.456024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.458563] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.462882] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.467055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.471000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.477017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.491023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.495725] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.501966] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.511027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.520037] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.539030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.550000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.562025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.571025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.590034] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.599000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.608034] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.613028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.625022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.633120] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.639028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.645027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.660036] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.670636] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.678021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.686023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.706029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.718961] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.729040] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.738032] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.758037] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.771222] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.773324] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.775328] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.776347] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.778256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.782297] iommu: Default domain type: Passthrough [ 0.785303] SCSI subsystem initialized [ 0.786149] ACPI: bus type USB registered [ 0.787099] usbcore: registered new interface driver usbfs [ 0.789081] usbcore: registered new interface driver hub [ 0.790062] usbcore: registered new device driver usb [ 0.791154] pps_core: LinuxPPS API ver. 1 registered [ 0.793014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.795074] PTP clock support registered [ 0.796125] EDAC MC: Ver: 3.0.0 [ 0.798137] PCI: Using ACPI for IRQ routing [ 0.799481] NetLabel: Initializing [ 0.800009] NetLabel: domain hash size = 128 [ 0.801008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.802079] NetLabel: unlabeled traffic allowed by default [ 0.804215] vgaarb: loaded [ 0.805263] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.806022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811368] clocksource: Switched to clocksource kvm-clock [ 0.917913] VFS: Disk quotas dquot_6.6.0 [ 0.919912] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.922919] *** VALIDATE ramfs *** [ 0.924510] *** VALIDATE hugetlbfs *** [ 0.926466] pnp: PnP ACPI init [ 0.929077] pnp: PnP ACPI: found 6 devices [ 0.946967] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.951124] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.953931] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.956656] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.959731] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.962594] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.965963] NET: Registered protocol family 2 [ 0.968747] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.974642] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.979328] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.986538] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.990440] TCP: Hash tables configured (established 65536 bind 65536) [ 0.993484] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.996895] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.000456] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.003723] NET: Registered protocol family 1 [ 1.006305] RPC: Registered named UNIX socket transport module. [ 1.008433] RPC: Registered udp transport module. [ 1.010154] RPC: Registered tcp transport module. [ 1.011612] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.013942] NET: Registered protocol family 44 [ 1.017165] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.018720] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.020301] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.022166] PCI: CLS 0 bytes, default 64 [ 1.023419] Unpacking initramfs... [ 2.545317] debug: unmapping init [mem 0xffff9a4abcc54000-0xffff9a4abffbffff] [ 2.550597] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.553895] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.558053] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.046313] Initialise system trusted keyrings [ 3.047509] Key type blacklist registered [ 3.049853] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.059269] zbud: loaded [ 3.062479] *** VALIDATE nfs *** [ 3.063383] *** VALIDATE nfs4 *** [ 3.064625] pstore: using deflate compression [ 3.067197] Platform Keyring initialized [ 3.183413] NET: Registered protocol family 38 [ 3.185024] Key type asymmetric registered [ 3.186227] Asymmetric key parser 'x509' registered [ 3.187609] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.189836] io scheduler mq-deadline registered [ 3.191068] io scheduler kyber registered [ 3.192432] io scheduler bfq registered [ 3.194150] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.196403] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.199194] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.201308] ACPI: Power Button [PWRF] [ 3.206265] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.214703] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.240327] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.257158] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.276430] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.301399] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.328413] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.334119] Non-volatile memory driver v1.3 [ 3.335811] Linux agpgart interface v0.103 [ 3.371151] virtio_blk virtio1: [vda] 132376 512-byte logical blocks (67.8 MB/64.6 MiB) [ 3.373456] vda: detected capacity change from 0 to 67776512 [ 3.391941] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.395203] vdb: detected capacity change from 0 to 1073741824 [ 3.414395] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.417714] vdc: detected capacity change from 0 to 2621440000 [ 3.437047] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.439989] vdd: detected capacity change from 0 to 2621440000 [ 3.457830] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.460121] vde: detected capacity change from 0 to 4294967296 [ 3.498022] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.500917] vdf: detected capacity change from 0 to 4294967296 [ 3.513641] libphy: Fixed MDIO Bus: probed [ 3.519203] usbcore: registered new interface driver usbserial_generic [ 3.521176] usbserial: USB Serial support registered for generic [ 3.523688] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.527961] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.530039] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.533602] mousedev: PS/2 mouse device common for all mice [ 3.537204] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.539378] rtc_cmos 00:05: RTC can wake from S4 [ 3.548879] rtc_cmos 00:05: registered as rtc0 [ 3.550695] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.554100] intel_pstate: CPU model not supported [ 3.555835] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.562093] hid: raw HID events driver (C) Jiri Kosina [ 3.567722] usbcore: registered new interface driver usbhid [ 3.570357] usbhid: USB HID core driver [ 3.572209] drop_monitor: Initializing network drop monitor service [ 3.575394] Initializing XFRM netlink socket [ 3.577360] NET: Registered protocol family 10 [ 3.580520] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.580887] Segment Routing with IPv6 [ 3.586132] NET: Registered protocol family 17 [ 3.588675] mpls_gso: MPLS GSO support [ 3.594796] RAS: Correctable Errors collector initialized. [ 3.596330] AVX version of gcm_enc/dec engaged. [ 3.597500] AES CTR mode by8 optimization enabled [ 3.693971] sched_clock: Marking stable (3693944588, 0)->(4659398314, -965453726) [ 3.699164] registered taskstats version 1 [ 3.703365] Loading compiled-in X.509 certificates [ 3.705880] zswap: loaded using pool lzo/zbud [ 3.733901] Key type big_key registered [ 3.751201] Key type encrypted registered [ 3.753401] ima: No TPM chip found, activating TPM-bypass! [ 3.756472] ima: Allocated hash algorithm: sha1 [ 3.758652] ima: No architecture policies found [ 3.760759] evm: Initialising EVM extended attributes: [ 3.763461] evm: security.selinux [ 3.765375] evm: security.ima [ 3.766495] evm: security.capability [ 3.768442] evm: HMAC attrs: 0x1 [ 3.772273] rtc_cmos 00:05: setting system clock to 2026-03-30 12:39:57 UTC (1774874397) [ 3.779245] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 3.782362] debug: unmapping init [mem 0xffffffffa8b82000-0xffffffffa8e58fff] [ 3.789182] Write protecting the kernel read-only data: 28672k [ 3.793586] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 3.796823] debug: unmapping init [mem 0xffffffffa7b14000-0xffffffffa7bfffff] [ 3.840454] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.847359] systemd[1]: Detected virtualization kvm. [ 3.848862] systemd[1]: Detected architecture x86-64. [ 3.850324] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.876776] systemd[1]: No hostname configured. [ 3.877957] systemd[1]: Set hostname to . [ 3.879728] random: systemd: uninitialized urandom read (16 bytes read) [ 3.882236] systemd[1]: Initializing machine ID from random generator. [ 3.944416] random: ln: uninitialized urandom read (6 bytes read) [ 4.041634] random: systemd: uninitialized urandom read (16 bytes read) [ 4.044348] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.052229] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.057200] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.795444] device-mapper: uevent: version 1.0.3 [ 4.797964] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.509471] virtio_net virtio0 ens2: renamed from eth0 [ 5.541942] random: fast init done [ 5.659337] scsi host0: ata_piix [ 5.669448] scsi host1: ata_piix [ 5.672701] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.675503] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.290039] random: crng init done [ 10.293451] random: 7 urandom warning(s) missed due to ratelimiting [ 11.025455] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.559933] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.702319] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.769778] SELinux: Disabled at runtime. [ 16.910996] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.937136] systemd[1]: Detected virtualization kvm. [ 16.947608] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.096889] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.107963] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.127364] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.143476] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.159385] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.176900] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.197384] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ 19.830784] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.917338] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.227345] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.483735] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.081062] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.145188] EDAC sbridge: Ver: 1.1.2 [ 27.108501] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 27.816605] NFS: Registering the id_resolver key type [ 27.818160] Key type id_resolver registered [ 27.819507] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg208-server login: [ 43.598169] hrtimer: interrupt took 2532106 ns [ 102.283250] libcfs: loading out-of-tree module taints kernel. [ 102.349153] Key type ._llcrypt registered [ 102.351628] Key type .llcrypt registered [ 102.509783] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 122.452942] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 124.558396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.598965] alg: No test for adler32 (adler32-zlib) [ 126.352885] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 127.466723] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 129.264259] Key type lgssc registered [ 131.873585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.193788] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 206.532978] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 224.384684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 224.429434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.844265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 225.876780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 225.926397] Lustre: lustre-MDT0000: new disk, initializing [ 225.966313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.975476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 230.397224] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.638942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 250.736812] Lustre: 6515:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 250.793693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 250.797323] Lustre: Skipped 1 previous similar message [ 250.918049] Lustre: lustre-MDT0001: new disk, initializing [ 251.052800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.125363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 251.140492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 257.199897] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.125565] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 274.737624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 275.033115] Lustre: lustre-OST0000: new disk, initializing [ 275.035886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 275.088696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 281.682621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 281.698047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 281.786314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 282.922144] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 302.864868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 303.007643] Lustre: lustre-OST0001: new disk, initializing [ 303.015800] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 303.068662] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 309.844444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 309.861300] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 309.985880] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 311.120429] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 326.811899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 335.979370] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 348.384897] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing check_logdir /tmp/testlogs/ [ 354.182736] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing yml_node [ 360.255403] Lustre: DEBUG MARKER: Client: 2.17.51.23 [ 363.745409] Lustre: DEBUG MARKER: MDS: 2.17.51.23 [ 367.050353] Lustre: DEBUG MARKER: OSS: 2.17.51.23 [ 368.779693] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 30 08:46:00 EDT 2026 [ 388.155807] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 389.717659] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 407.522398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.532839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 407.536725] Lustre: Skipped 3 previous similar messages [ 412.643021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 417.762038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 417.776237] Lustre: Skipped 6 previous similar messages [ 420.320617] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 420.544876] Lustre: server umount lustre-MDT0000 complete [ 425.639636] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774874819 with bad export cookie 1858284457287088930 [ 425.644816] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 425.652392] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 428.015997] LustreError: 6520:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.025917] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.040416] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 428.042861] LustreError: 6520:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 428.081166] Lustre: Skipped 4 previous similar messages [ 429.025695] LustreError: 6521:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.055791] LustreError: 6521:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 430.052293] LustreError: 6520:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.062543] LustreError: 6520:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 432.284719] Lustre: server umount lustre-MDT0001 complete [ 437.203522] Lustre: server umount lustre-OST0000 complete [ 441.639330] Lustre: server umount lustre-OST0001 complete [ 449.195414] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 458.185410] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 498.406665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.617720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 498.642547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 498.722665] Lustre: lustre-MDT0000: new disk, initializing [ 498.856795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.875978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 504.216697] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 519.348316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.655285] Lustre: 15000:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 519.801461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 519.811069] Lustre: Skipped 1 previous similar message [ 520.021186] Lustre: lustre-MDT0001: new disk, initializing [ 520.205230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 520.262893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 520.280277] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 525.458308] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 534.604688] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 540.568920] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 551.633405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 552.295596] Lustre: lustre-OST0000: new disk, initializing [ 552.314296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 552.562790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.034552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 554.044209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 554.296333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 561.002884] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 571.212891] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 576.897947] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 577.252442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 583.084741] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 583.365723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 590.306173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.314540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 590.335519] Lustre: Skipped 1 previous similar message [ 590.342790] Lustre: Skipped 5 previous similar messages [ 592.314188] Lustre: server umount lustre-OST0000 complete [ 600.547385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.564929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 600.570542] Lustre: Skipped 1 previous similar message [ 600.583962] Lustre: Skipped 2 previous similar messages [ 603.832548] Lustre: server umount lustre-MDT0000 complete [ 605.669136] LustreError: 15818:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.701605] LustreError: 15818:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 608.811054] LustreError: 14990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774875002 with bad export cookie 1858284457287090456 [ 608.825898] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.828624] LustreError: 14990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 609.576991] Lustre: server umount lustre-MDT0001 complete [ 614.662806] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:50:05 (1774875005) [ 617.635720] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 619.554791] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:50:10 (1774875010) [ 627.815146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.471250] LustreError: 19005:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.561291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.561495] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 638.949058] LustreError: 19006:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.986153] LustreError: 19006:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 643.478324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.575476] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 657.204508] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 661.492235] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 669.796889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 678.820303] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 685.798662] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 698.850935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 698.860043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.877104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 698.912044] Lustre: Skipped 1 previous similar message [ 698.920487] Lustre: Skipped 1 previous similar message [ 703.257746] Lustre: server umount lustre-OST0000 complete [ 709.107553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.132912] Lustre: Skipped 1 previous similar message [ 722.400623] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 722.700909] Lustre: server umount lustre-MDT0000 complete [ 723.430928] LustreError: 19716:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 723.452854] LustreError: 19716:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 727.630372] LustreError: 18986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774875121 with bad export cookie 1858284457287091338 [ 727.635893] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.643778] LustreError: 18986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 728.830046] Lustre: server umount lustre-MDT0001 complete [ 740.529480] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 744.421607] Key type lgssc unregistered [ 744.901441] LNet: 22191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.917131] LNetError: 22191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.962404] LNet: Removed LNI 192.168.202.108@tcp [ 747.279527] Key type .llcrypt unregistered [ 747.284673] Key type ._llcrypt unregistered [ 775.946723] Key type ._llcrypt registered [ 775.950359] Key type .llcrypt registered [ 776.203899] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 777.544958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.728622] alg: No test for adler32 (adler32-zlib) [ 779.271285] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 779.756183] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 781.544341] Key type lgssc registered [ 782.885294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 793.605992] loop: module loaded [ 793.713966] loop0: detected capacity change from 0 to 204800000 [ 794.029858] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 794.066317] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 794.662540] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 801.411289] loop0: detected capacity change from 0 to 204800000 [ 801.619647] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 801.632201] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 801.648858] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 807.693835] loop0: detected capacity change from 0 to 204800000 [ 813.814557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 813.844977] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.289528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 815.368987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 815.492532] Lustre: lustre-MDT0000: new disk, initializing [ 815.693099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.735662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 820.835663] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 832.292647] loop1: detected capacity change from 0 to 204800000 [ 838.326754] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 838.626839] Lustre: lustre-OST07c6: new disk, initializing [ 838.633862] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 838.642249] Lustre: Skipped 1 previous similar message [ 838.719711] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 844.416350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 844.433435] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 844.484531] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 846.298747] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 852.738401] Lustre: 24763:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 852.757165] Lustre: 24763:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 852.777722] Lustre: 24763:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 852.790032] Lustre: 24763:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 852.798989] Lustre: 24763:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 852.804489] Lustre: 24763:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 866.803919] Lustre: 24346:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 866.811984] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 866.818450] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 866.823680] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 866.829899] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 866.834815] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 872.373177] Lustre: 24346:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 872.392588] Lustre: 24346:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 872.419218] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 872.433120] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.448439] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 872.454484] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.467330] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 872.480715] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.498765] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 872.510117] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.520414] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 872.533319] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 877.982851] Lustre: 24763:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 877.990338] Lustre: 24763:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 877.995725] Lustre: 24763:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 878.010820] Lustre: 24763:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 878.021773] Lustre: 24763:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 878.029459] Lustre: 24763:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 878.034157] Lustre: 24763:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 878.041465] Lustre: 24763:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 878.047682] Lustre: 24763:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 878.054326] Lustre: 24763:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 878.060623] Lustre: 24763:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 878.067346] Lustre: 24763:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 886.089680] Lustre: 24763:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 886.103294] Lustre: 24763:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages [ 886.111425] Lustre: 24763:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 886.116988] Lustre: 24763:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 886.122220] Lustre: 24763:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 886.135320] Lustre: 24763:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 886.151244] Lustre: 24763:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 886.162354] Lustre: 24763:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 886.174685] Lustre: 24763:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 886.185249] Lustre: 24763:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 886.190863] Lustre: 24763:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 886.200507] Lustre: 24763:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 897.979773] Lustre: 24346:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 897.996343] Lustre: 24346:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages [ 898.004213] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 898.037920] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 898.060134] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 898.068759] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 898.079623] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/1 [ 898.095917] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 898.110631] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 898.121560] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 898.136016] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 898.144231] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 947.169294] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.178149] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.188140] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 952.298322] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 956.384301] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 956.643232] Lustre: server umount lustre-OST07c6 complete [ 967.680977] Lustre: server umount lustre-MDT0000 complete [ 985.612395] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 988.995137] Key type lgssc unregistered [ 989.254934] LNet: 27874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.273597] LNetError: 27874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.306503] LNet: Removed LNI 192.168.202.108@tcp [ 990.134423] Key type .llcrypt unregistered [ 990.137231] Key type ._llcrypt unregistered [ 1008.742380] Key type ._llcrypt registered [ 1008.744771] Key type .llcrypt registered [ 1008.899476] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1009.795385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1009.862521] alg: No test for adler32 (adler32-zlib) [ 1010.940866] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 1011.143111] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1012.784176] Key type lgssc registered [ 1014.601722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1023.260042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1023.276336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.863104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.940695] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1035.256589] LustreError: 29130:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.281338] LustreError: 29130:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1038.561344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.035757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1043.315302] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1049.412937] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1053.912755] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1061.111356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1061.793597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.757972] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1070.118560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1075.712431] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1090.533595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1090.536565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.568811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.577665] Lustre: Skipped 1 previous similar message [ 1092.271719] Lustre: server umount lustre-OST0000 complete [ 1100.769756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1100.775641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.779327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1100.793699] Lustre: Skipped 2 previous similar messages [ 1100.805415] Lustre: Skipped 1 previous similar message [ 1102.729460] Lustre: server umount lustre-MDT0000 complete [ 1105.901983] LustreError: 29125:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.932308] LustreError: 29125:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1107.305444] LustreError: 29110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774875501 with bad export cookie 15689882624843796993 [ 1107.321802] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.323056] LustreError: 29110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1107.804836] Lustre: server umount lustre-MDT0001 complete [ 1119.717654] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1123.060430] Key type lgssc unregistered [ 1123.544688] LNet: 32515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.555666] LNetError: 32515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.595613] LNet: Removed LNI 192.168.202.108@tcp [ 1124.579345] Key type .llcrypt unregistered [ 1124.585422] Key type ._llcrypt unregistered [ 1143.316606] Key type ._llcrypt registered [ 1143.319492] Key type .llcrypt registered [ 1143.412883] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1144.714320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.784520] alg: No test for adler32 (adler32-zlib) [ 1145.915472] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 1146.141586] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1147.784179] Key type lgssc registered [ 1148.931417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1158.866307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1158.947788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.780556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.162845] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1170.919253] LustreError: 33766:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.938959] LustreError: 33766:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1176.038383] LustreError: 33767:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.662200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.989548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1182.107668] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1189.212765] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.689847] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1203.188965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1203.557230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.658182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1210.568151] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1217.621657] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1235.525685] Lustre: server umount lustre-OST0000 complete [ 1242.594848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.604658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.613439] Lustre: Skipped 1 previous similar message [ 1247.722899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.732251] Lustre: Skipped 2 previous similar messages [ 1250.907032] Lustre: server umount lustre-MDT0000 complete [ 1252.834298] LustreError: 33767:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.857357] LustreError: 33767:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1253.792220] Lustre: 33080:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774875631/real 1774875631] req@00000000a78861f5 x1861091893212288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774875647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1253.818521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.270387] LustreError: 33747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774875648 with bad export cookie 1219102022471005664 [ 1255.293513] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.853134] Lustre: server umount lustre-MDT0001 complete [ 1267.031743] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1271.854414] Key type lgssc unregistered [ 1272.289763] LNet: 36984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.293864] LNetError: 36984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.305324] LNet: Removed LNI 192.168.202.108@tcp [ 1273.408749] Key type .llcrypt unregistered [ 1273.414996] Key type ._llcrypt unregistered [ 1298.027522] Key type ._llcrypt registered [ 1298.035403] Key type .llcrypt registered [ 1298.167651] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1299.053173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1299.189812] alg: No test for adler32 (adler32-zlib) [ 1300.444054] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 1300.697249] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1302.384388] Key type lgssc registered [ 1303.683736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1312.289051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1312.317596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.169136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.464665] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1323.494620] LustreError: 38400:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.532316] LustreError: 38400:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1328.610386] LustreError: 38399:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.198689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.779777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.775591] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1342.229967] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1346.480638] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1353.534034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1353.926459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.109924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1361.105329] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1368.060421] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1375.829217] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 1376.740388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1376.756430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.783852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.810486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.815554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.850038] Lustre: Skipped 2 previous similar messages [ 1381.837705] Lustre: server umount lustre-MDT0000 complete [ 1384.941147] LustreError: 38396:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.970205] LustreError: 38396:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1385.969889] LustreError: 38381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774875779 with bad export cookie 13632972094429999808 [ 1385.995376] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1386.014552] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [ 1386.018813] Lustre: Skipped 1 previous similar message [ 1390.052162] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.055368] LustreError: 38394:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.059275] Lustre: Skipped 1 previous similar message [ 1390.081757] LustreError: 38394:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1390.100361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1395.170761] LustreError: 38396:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.194153] LustreError: 38396:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1400.288193] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1400.563527] Lustre: server umount lustre-MDT0001 complete [ 1416.340807] Lustre: server umount lustre-OST0000 complete [ 1434.199608] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1437.399876] Key type lgssc unregistered [ 1437.725301] LNet: 41908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.737150] LNetError: 41908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.758497] LNet: Removed LNI 192.168.202.108@tcp [ 1438.721437] Key type .llcrypt unregistered [ 1438.723757] Key type ._llcrypt unregistered [ 1468.579746] Key type ._llcrypt registered [ 1468.583347] Key type .llcrypt registered [ 1468.723879] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1470.039959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1470.052764] alg: No test for adler32 (adler32-zlib) [ 1471.153701] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 1471.438187] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1473.112179] Key type lgssc registered [ 1474.716304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1483.238603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1483.275267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.867360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.035308] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1495.020397] LustreError: 43311:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.046792] LustreError: 43311:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1499.324510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1499.638507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1504.240776] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1510.876533] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1515.790479] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1523.987979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1524.511803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.687473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1531.873213] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1539.049054] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1545.190820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.205244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1545.229694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1545.761244] Lustre: lustre-OST0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 1545.766622] Lustre: Skipped 1 previous similar message [ 1550.325778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1550.863514] LustreError: 44906:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1551.135449] Lustre: server umount lustre-OST0000 complete [ 1570.792710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.810154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.816566] Lustre: Skipped 2 previous similar messages [ 1570.841478] Lustre: Skipped 2 previous similar messages [ 1575.909657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.935282] Lustre: Skipped 1 previous similar message [ 1577.241695] Lustre: server umount lustre-MDT0000 complete [ 1582.050859] LustreError: 45369:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.079670] LustreError: 45369:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1582.651294] LustreError: 43290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774875976 with bad export cookie 13180608480261298125 [ 1582.654233] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.665353] LustreError: 43290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1589.223414] Lustre: server umount lustre-MDT0001 complete [ 1600.887896] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1603.851374] Key type lgssc unregistered [ 1604.127748] LNet: 46670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.134173] LNetError: 46670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1604.161686] LNet: Removed LNI 192.168.202.108@tcp [ 1604.817738] Key type .llcrypt unregistered [ 1604.819831] Key type ._llcrypt unregistered [ 1627.351536] Key type ._llcrypt registered [ 1627.354661] Key type .llcrypt registered [ 1627.497235] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1628.852248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.917030] alg: No test for adler32 (adler32-zlib) [ 1630.157895] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 1630.507549] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1632.218163] Key type lgssc registered [ 1633.955305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1642.784894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1642.812572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.410201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.587734] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1654.804192] LustreError: 48071:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.841814] LustreError: 48071:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1659.467775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.902385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1665.164479] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1671.925885] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1675.908552] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1684.960031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1685.475887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.611466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1693.511611] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1699.802649] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1709.677063] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1713.258895] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1726.069959] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-30 9h08m36s [ 1727.976226] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1730.535464] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.541947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.560492] Lustre: Skipped 1 previous similar message [ 1731.618709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1731.634382] Lustre: Skipped 2 previous similar messages [ 1733.689226] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [ 1736.655418] Lustre: server umount lustre-MDT0000 complete [ 1737.711663] LustreError: 48065:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.743099] LustreError: 48065:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1741.215816] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1741.317123] LustreError: 48065:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.850468] LustreError: 48070:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.879692] LustreError: 48070:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1748.656243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1748.796727] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.083233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.141610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1750.134928] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1753.456661] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1757.659773] Lustre: DEBUG MARKER: Wait for df (36132) ... [ 1759.595688] Lustre: DEBUG MARKER: done [ 1765.871468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.876520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.884892] Lustre: Skipped 2 previous similar messages [ 1765.894631] Lustre: Skipped 2 previous similar messages [ 1770.978193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.991022] Lustre: Skipped 1 previous similar message [ 1779.168136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1779.288637] Lustre: server umount lustre-OST0000 complete [ 1786.342840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.348966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1786.371105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.371115] Lustre: Skipped 2 previous similar messages [ 1790.043581] Lustre: server umount lustre-MDT0000 complete [ 1793.507098] LustreError: 48782:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.542082] LustreError: 48782:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1794.994285] LustreError: 48051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774876188 with bad export cookie 8545357366854230700 [ 1795.010982] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1795.012819] LustreError: 48051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1795.574506] Lustre: server umount lustre-MDT0001 complete [ 1807.660952] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 1811.211758] Key type lgssc unregistered [ 1811.502579] LNet: 52631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.507399] LNetError: 52631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.525568] LNet: Removed LNI 192.168.202.108@tcp [ 1812.549769] Key type .llcrypt unregistered [ 1812.554560] Key type ._llcrypt unregistered [ 1861.736688] Key type ._llcrypt registered [ 1861.739657] Key type .llcrypt registered [ 1861.934285] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 1863.997778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.009205] alg: No test for adler32 (adler32-zlib) [ 1865.241829] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 1865.661624] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 1867.480340] Key type lgssc registered [ 1869.340149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1878.860468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1878.921348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.456681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1880.513534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1880.717296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.189824] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1894.109316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.290513] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1894.336261] Lustre: 54829:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1894.862586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1899.669771] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1905.276790] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1909.333458] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1916.080639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1916.338594] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1916.605075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.208893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1923.351616] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 1929.637781] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1937.266519] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-30 9h12m08s [ 1938.959945] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1947.318435] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1950.339348] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1950.412833] LustreError: 54837:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1970.735100] Lustre: lustre-MDT0000: Client 929976f5-68c1-4cd3-acb8-8e9c8f84663e (at 192.168.202.8@tcp) reconnecting [ 1990.498625] LustreError: 54837:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1993.063508] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1996.213436] Lustre: DEBUG MARKER: done [ 2003.938071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.944947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.978391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2004.451306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2004.465250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.481543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2004.486495] Lustre: Skipped 1 previous similar message [ 2009.572658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.580950] Lustre: Skipped 1 previous similar message [ 2010.168329] Lustre: server umount lustre-OST0000 complete [ 2017.773415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2017.776702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.801721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.355374] Lustre: server umount lustre-MDT0000 complete [ 2025.747623] LustreError: 56465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774876419 with bad export cookie 17989787562514101178 [ 2025.748850] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.759949] LustreError: 56465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2026.233713] Lustre: server umount lustre-MDT0001 complete [ 2038.523572] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2041.671344] Key type lgssc unregistered [ 2042.099935] LNet: 58802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.104827] LNetError: 58802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.145464] LNet: Removed LNI 192.168.202.108@tcp [ 2043.117084] Key type .llcrypt unregistered [ 2043.119643] Key type ._llcrypt unregistered [ 2092.480293] Key type ._llcrypt registered [ 2092.483040] Key type .llcrypt registered [ 2092.583283] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2094.154880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2094.215075] alg: No test for adler32 (adler32-zlib) [ 2095.325760] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 2095.540303] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2097.208475] Key type lgssc registered [ 2098.519907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2102.375614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2102.408667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.897494] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2103.939531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2104.105578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.974477] Lustre: Failing over lustre-MDT0000 [ 2105.175424] Lustre: server umount lustre-MDT0000 complete [ 2112.348848] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:15:03 (1774876503) [ 2119.707299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.146920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.280719] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2135.480871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.593826] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2135.628159] Lustre: 61588:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2136.032402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2140.839212] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2147.814937] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2152.871408] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2160.705933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.963568] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2161.153249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.463267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2168.829558] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2175.608431] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2181.453444] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2192.353637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.357073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.386680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.380213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.381847] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.398529] Lustre: Skipped 1 previous similar message [ 2196.259285] Lustre: server umount lustre-MDT0000 complete [ 2200.184096] LustreError: 61578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774876593 with bad export cookie 2322015674295332831 [ 2200.187362] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.211914] LustreError: 61578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2200.776215] Lustre: server umount lustre-MDT0001 complete [ 2204.332467] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2208.877104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2218.854223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.064209] Lustre: 60512:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774876597/real 1774876597] req@00000000498969a3 x1861092889371520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774876613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2220.098801] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.048204] Lustre: 60509:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774876599/real 1774876599] req@0000000023489a9e x1861092889371648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774876615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2225.184990] Lustre: 60509:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774876602/real 1774876602] req@000000009c45a755 x1861092889371776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774876618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2225.249090] LustreError: 60508:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007f2eeefd x1861092889372672/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2225.719738] LustreError: 63207:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.743633] LustreError: 63207:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2225.874341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.982459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2229.924287] LustreError: 63208:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.943104] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2230.945108] Lustre: 60512:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774876608/real 1774876608] req@00000000d9836b49 x1861092889372160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774876624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2230.961495] LustreError: 65020:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.979720] Lustre: 60512:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2231.010472] LustreError: 65020:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2236.064164] Lustre: 60509:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774876613/real 1774876613] req@00000000e8d75720 x1861092889372288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774876629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2236.388105] LustreError: 65019:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.430748] LustreError: 65019:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2240.442051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.151034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.198279] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2246.635781] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2248.719535] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2253.386523] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2258.780593] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2269.153445] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.159039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.171186] Lustre: Skipped 1 previous similar message [ 2269.174859] Lustre: Skipped 2 previous similar messages [ 2273.830145] Lustre: server umount lustre-MDT0000 complete [ 2274.280031] LustreError: 65019:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.317324] LustreError: 65019:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2277.730410] LustreError: 61579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774876671 with bad export cookie 2322015674295335106 [ 2277.747774] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.974414] Lustre: server umount lustre-MDT0001 complete [ 2284.724125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2295.460490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.520092] Lustre: 60510:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774876673/real 1774876673] req@000000005617ca80 x1861092889400576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774876689 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2295.546742] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.561710] Lustre: Skipped 1 previous similar message [ 2302.754348] LustreError: 60508:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000adde506f x1861092889401600/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2303.021342] LustreError: 63207:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.098913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.152987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2307.874666] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2317.266735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.694632] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2322.930869] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2325.039209] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2329.430368] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2333.723519] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2343.398041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2343.401566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.425247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.440301] Lustre: Skipped 2 previous similar messages [ 2350.563333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2350.575713] Lustre: Skipped 5 previous similar messages [ 2355.168124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2355.399268] Lustre: server umount lustre-MDT0000 complete [ 2355.697178] LustreError: 68648:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.715555] LustreError: 68648:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 2359.251331] LustreError: 61580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774876752 with bad export cookie 2322015674295336485 [ 2359.258119] LustreError: 61580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2359.268342] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.588774] Lustre: server umount lustre-MDT0001 complete [ 2366.634804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2371.513827] Lustre: DEBUG MARKER: files compared the same [ 2384.167780] Lustre: server umount lustre-OST0000 complete [ 2401.390989] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2404.126812] Key type lgssc unregistered [ 2404.525443] LNet: 70518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2404.538349] LNetError: 70518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2404.565341] LNet: Removed LNI 192.168.202.108@tcp [ 2405.503708] Key type .llcrypt unregistered [ 2405.505799] Key type ._llcrypt unregistered [ 2426.969885] Key type ._llcrypt registered [ 2426.971721] Key type .llcrypt registered [ 2427.082535] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2428.567064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2428.692714] alg: No test for adler32 (adler32-zlib) [ 2429.972269] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 2430.322538] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2432.008348] Key type lgssc registered [ 2433.193943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2442.268655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2442.300680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.815847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.950771] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug +malloc all [ 2453.989445] LustreError: 71897:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.000433] LustreError: 71897:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2458.578158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.793331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2462.553073] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug +malloc all [ 2470.341354] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2477.631979] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2489.090688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.466094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.260673] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug +malloc all [ 2504.497805] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2514.041092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2524.641785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2524.649119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.653192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2524.679057] Lustre: Skipped 1 previous similar message [ 2529.761744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.777043] Lustre: Skipped 2 previous similar messages [ 2529.939184] Lustre: server umount lustre-OST0000 complete [ 2534.893263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.905365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.910080] Lustre: Skipped 1 previous similar message [ 2534.927262] Lustre: Skipped 1 previous similar message [ 2540.004500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.009079] Lustre: Skipped 1 previous similar message [ 2545.121587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.138681] Lustre: Skipped 1 previous similar message [ 2549.216169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2550.244813] LustreError: 71897:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.273454] LustreError: 71897:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2551.399855] Lustre: server umount lustre-MDT0000 complete [ 2554.354342] LustreError: 71898:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.374724] LustreError: 71898:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2556.506972] LustreError: 71877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774876950 with bad export cookie 13868701794159408982 [ 2556.520307] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.188244] Lustre: server umount lustre-MDT0001 complete [ 2570.836830] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2574.303460] Key type lgssc unregistered [ 2574.619168] LNet: 75102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2574.633084] LNetError: 75102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2574.661711] LNet: Removed LNI 192.168.202.108@tcp [ 2575.711849] Key type .llcrypt unregistered [ 2575.714837] Key type ._llcrypt unregistered [ 2603.529688] Key type ._llcrypt registered [ 2603.534165] Key type .llcrypt registered [ 2603.755074] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2605.448454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2605.639742] alg: No test for adler32 (adler32-zlib) [ 2606.870312] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 2607.198837] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2608.864211] Key type lgssc registered [ 2610.175992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2620.217507] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2620.263872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.065750] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2641.184558] LustreError: 76478:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [ 2667.939351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2676.648775] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2685.638488] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2697.016969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.147795] Lustre: *** cfs_fail_loc=706, val=0*** [ 2697.724852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.955185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2702.821710] LustreError: 77452:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.848837] LustreError: 77452:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2704.752730] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2707.939972] LustreError: 77451:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.953140] LustreError: 77451:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2713.065771] LustreError: 77453:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.103930] LustreError: 77453:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2715.580298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.486373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.809778] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2727.871084] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2732.994225] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2737.121775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.128385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.137232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.158788] Lustre: Skipped 1 previous similar message [ 2741.567637] Lustre: server umount lustre-OST0000 complete [ 2747.388028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.392349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.395197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2747.401876] Lustre: Skipped 1 previous similar message [ 2747.419729] Lustre: Skipped 2 previous similar messages [ 2752.481811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2752.679492] Lustre: server umount lustre-MDT0000 complete [ 2756.582509] LustreError: 79007:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.601515] LustreError: 79007:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2756.746955] LustreError: 76486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774877150 with bad export cookie 4242848312797297963 [ 2756.757719] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2757.031113] Lustre: server umount lustre-MDT0001 complete [ 2769.422588] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 2773.592346] Key type lgssc unregistered [ 2773.854907] LNet: 79741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2773.861064] LNetError: 79741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2773.879258] LNet: Removed LNI 192.168.202.108@tcp [ 2775.107989] Key type .llcrypt unregistered [ 2775.109925] Key type ._llcrypt unregistered [ 2803.443491] Key type ._llcrypt registered [ 2803.446598] Key type .llcrypt registered [ 2803.623874] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2804.728867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2804.815221] alg: No test for adler32 (adler32-zlib) [ 2806.109891] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 2806.452354] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 2808.201939] Key type lgssc registered [ 2809.463848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2817.566534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2817.583647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.616627] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2833.101911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.969596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2838.834867] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2839.020604] LustreError: 81812:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.032165] LustreError: 81812:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 2844.129995] LustreError: 81816:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.148778] LustreError: 81816:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2847.535644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2847.784877] LustreError: 82077:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.798570] LustreError: 82077:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2847.834578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.850771] LustreError: 81811:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.881780] LustreError: 81811:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2853.769825] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2857.964879] LustreError: 81810:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.986627] LustreError: 81810:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2862.421711] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.435916] Lustre: MGS: Client 5663735d-474a-4298-a4ad-cf3b749b260d (at 0@lo) reconnecting [ 2862.468899] LustreError: 81827:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2862.660210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.759184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2866.999426] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 2883.053233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.067602] Lustre: Skipped 1 previous similar message [ 2883.071364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2883.076066] Lustre: Skipped 1 previous similar message [ 2886.594837] Lustre: server umount lustre-OST0000 complete [ 2893.281869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.288882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.291932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.310436] Lustre: Skipped 1 previous similar message [ 2893.326661] Lustre: Skipped 1 previous similar message [ 2896.862194] Lustre: server umount lustre-MDT0000 complete [ 2898.405208] LustreError: 83326:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.429018] LustreError: 83326:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 2901.299311] Lustre: server umount lustre-MDT0001 complete [ 2905.210129] Lustre: server umount MGS complete [ 2916.640364] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:28:28 (1774877308) [ 2947.094305] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 2956.412205] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 2992.315932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.103430] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3009.347238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.705304] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3009.873943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3009.994795] Lustre: lustre-MDT0001: new disk, initializing [ 3010.183591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3010.213678] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3010.223364] Lustre: 88888:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3010.230788] LustreError: 88888:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3015.088969] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3026.203343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.498484] Lustre: lustre-OST0000: new disk, initializing [ 3026.507308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3026.601841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.808616] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3034.211338] Lustre: 89923:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000086423f9b x1861093634907776/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1774877443 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3041.178834] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3048.650732] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.672270] Lustre: MGS: Client 99e7ffb1-a7c1-4b02-a230-739fa4c5ffc4 (at 0@lo) reconnecting [ 3048.702312] LustreError: 88874:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3048.738560] Lustre: 90570:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3048.789877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3048.856388] Lustre: lustre-MDT0000: new disk, initializing [ 3048.992221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.013600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3049.035613] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3050.088286] Lustre: 90596:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000040f4e4d8 x1861093634914432/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1774877459 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3054.114118] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3055.329956] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3059.252592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3059.262311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3059.418384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3087.330925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3087.352098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.359613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.378131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.381282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.721145] Lustre: server umount lustre-OST0000 complete [ 3099.623904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.627984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3099.642755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.642764] Lustre: Skipped 1 previous similar message [ 3099.651367] Lustre: Skipped 1 previous similar message [ 3102.749127] Lustre: server umount lustre-MDT0000 complete [ 3104.740594] LustreError: 91002:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.764846] LustreError: 91002:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3114.050181] Lustre: server umount lustre-MDT0001 complete [ 3118.417820] Lustre: server umount MGS complete [ 3132.012974] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:32:03 (1774877523) [ 3150.105554] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3153.554309] Key type lgssc unregistered [ 3153.904391] LNet: 93062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3153.915605] LNetError: 93062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3154.983738] LNet: Removed LNI 192.168.202.108@tcp [ 3156.106547] Key type .llcrypt unregistered [ 3156.108418] Key type ._llcrypt unregistered [ 3173.814559] Key type ._llcrypt registered [ 3173.824267] Key type .llcrypt registered [ 3173.967957] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3175.143982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3175.230527] alg: No test for adler32 (adler32-zlib) [ 3176.265306] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 3176.492526] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3178.201560] Key type lgssc registered [ 3179.433911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3187.406326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3187.442377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.901103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.611562] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3198.446269] LustreError: 94271:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.485657] LustreError: 94271:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3202.445604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.900193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3207.812662] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3215.142625] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3219.658655] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3227.213537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.725592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.918482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3234.933547] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3241.556526] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3254.753399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3254.770863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.777933] LustreError: Skipped 1 previous similar message [ 3254.782952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.814942] Lustre: Skipped 1 previous similar message [ 3256.309808] Lustre: server umount lustre-OST0000 complete [ 3264.483609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3264.486751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.507203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3264.509573] Lustre: Skipped 1 previous similar message [ 3267.077131] Lustre: server umount lustre-MDT0000 complete [ 3270.125024] LustreError: 94267:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.153707] LustreError: 94267:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3271.155074] LustreError: 94252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774877664 with bad export cookie 8824199343064335548 [ 3271.159821] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3271.162253] LustreError: 94252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3271.518946] Lustre: server umount lustre-MDT0001 complete [ 3282.990636] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3285.963654] Key type lgssc unregistered [ 3286.313950] LNet: 97451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3286.323821] LNetError: 97451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3286.342181] LNet: Removed LNI 192.168.202.108@tcp [ 3287.388370] Key type .llcrypt unregistered [ 3287.390658] Key type ._llcrypt unregistered [ 3305.223633] Key type ._llcrypt registered [ 3305.232572] Key type .llcrypt registered [ 3305.389409] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3307.007933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3307.090569] alg: No test for adler32 (adler32-zlib) [ 3308.316187] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 3308.612086] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3310.352180] Key type lgssc registered [ 3311.810479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3319.295729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3319.310740] LustreError: 98507:0:(tgt_mount.c:2521:server_fill_super()) cfs_race id 716 sleeping [ 3319.398982] LustreError: 98517:0:(tgt_mount.c:2521:server_fill_super()) cfs_fail_race id 716 waking [ 3319.409678] LustreError: 98507:0:(tgt_mount.c:2521:server_fill_super()) cfs_fail_race id 716 awake: rc=4910 [ 3319.423765] LustreError: 98507:0:(tgt_mount.c:2533:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3319.430689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.437742] LustreError: 98507:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3321.171366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.120727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.369357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3335.253615] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3341.873096] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing lsmod [ 3345.557463] LustreError: 99686:0:(tgt_mount.c:2521:server_fill_super()) cfs_race id 716 sleeping [ 3345.621866] LustreError: 99690:0:(tgt_mount.c:2521:server_fill_super()) cfs_fail_race id 716 waking [ 3345.636461] LustreError: 99686:0:(tgt_mount.c:2521:server_fill_super()) cfs_fail_race id 716 awake: rc=4934 [ 3345.644432] LustreError: 99686:0:(tgt_mount.c:2533:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3345.658746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.661103] LustreError: 99686:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3346.182672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.271556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3351.010246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3351.019503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3351.030569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.547517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.549464] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.580463] Lustre: Skipped 1 previous similar message [ 3354.660939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3354.678098] Lustre: Skipped 4 previous similar messages [ 3355.827269] Lustre: server umount lustre-MDT0000 complete [ 3359.718975] LustreError: 99291:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.751022] LustreError: 99291:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 3360.231509] LustreError: 98531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774877753 with bad export cookie 12105472490462240171 [ 3360.249787] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3360.252779] LustreError: 98531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3360.824301] LustreError: 98986:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000002a09f04 x1861094161275648/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3361.022897] Lustre: server umount lustre-MDT0001 complete [ 3376.258889] Lustre: server umount lustre-OST0000 complete [ 3385.230301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.508531] LustreError: 100758:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.574223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.948240] LustreError: 100759:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3391.086270] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3396.071208] LustreError: 100758:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.191105] LustreError: 100759:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.854329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.377622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3407.775558] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3414.784996] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3420.181843] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3429.365090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.987709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3431.031935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3437.722903] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3447.905357] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3456.993941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3457.004717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.023828] LustreError: Skipped 1 previous similar message [ 3457.026675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3457.061337] Lustre: Skipped 1 previous similar message [ 3462.123418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3462.126481] Lustre: Skipped 2 previous similar messages [ 3462.996259] Lustre: server umount lustre-OST0000 complete [ 3468.771651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3468.779445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.356280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3472.386395] Lustre: Skipped 2 previous similar messages [ 3473.885520] Lustre: server umount lustre-MDT0000 complete [ 3477.475451] LustreError: 100755:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3477.507399] LustreError: 100755:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3478.337284] LustreError: 100737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774877872 with bad export cookie 12105472490462241102 [ 3478.346533] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.807438] Lustre: server umount lustre-MDT0001 complete [ 3491.874912] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3495.447548] Key type lgssc unregistered [ 3495.760306] LNet: 103940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3495.765712] LNetError: 103940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3495.802039] LNet: Removed LNI 192.168.202.108@tcp [ 3496.929240] Key type .llcrypt unregistered [ 3496.931985] Key type ._llcrypt unregistered [ 3522.036960] Key type ._llcrypt registered [ 3522.039935] Key type .llcrypt registered [ 3522.175754] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3523.714191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3523.956842] alg: No test for adler32 (adler32-zlib) [ 3525.097506] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 3525.329221] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3527.056187] Key type lgssc registered [ 3528.590036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3537.902412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3537.933391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.529979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.708796] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3549.667988] LustreError: 105344:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.695575] LustreError: 105344:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3554.785075] LustreError: 105343:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.852540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.460576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3561.087207] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3568.987556] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3576.346208] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3585.422628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.009763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.158265] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3600.385841] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3609.605623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3617.560773] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3623.110995] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3630.051282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3630.063320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.070788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3630.111704] Lustre: Skipped 1 previous similar message [ 3634.609619] Lustre: server umount lustre-OST0000 complete [ 3640.305990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.329170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3640.339989] Lustre: Skipped 1 previous similar message [ 3640.366960] Lustre: Skipped 2 previous similar messages [ 3645.424381] LustreError: 105343:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.441028] LustreError: 105343:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3646.045109] Lustre: server umount lustre-MDT0000 complete [ 3649.518904] LustreError: 107574:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.546219] LustreError: 107574:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3651.215505] LustreError: 106960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774878044 with bad export cookie 16193853917013430907 [ 3651.217286] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.225880] LustreError: 106960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3651.732056] Lustre: server umount lustre-MDT0001 complete [ 3668.024235] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3672.329240] Key type lgssc unregistered [ 3672.648597] LNet: 108657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3672.655428] LNetError: 108657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3672.684744] LNet: Removed LNI 192.168.202.108@tcp [ 3673.657995] Key type .llcrypt unregistered [ 3673.660600] Key type ._llcrypt unregistered [ 3694.427458] Key type ._llcrypt registered [ 3694.439077] Key type .llcrypt registered [ 3694.602313] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3696.250489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3696.369502] alg: No test for adler32 (adler32-zlib) [ 3697.514992] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 3697.819467] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3699.488196] Key type lgssc registered [ 3701.031246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3709.914079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3709.939589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.605212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.613915] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3722.215402] LustreError: 109867:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3722.241781] LustreError: 109867:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3727.332463] LustreError: 109866:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.409520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3729.910479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3734.255023] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3742.205624] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3747.097442] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3756.720122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3757.394615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.782687] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3770.944904] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3782.022908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3792.353851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3792.358854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.373349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3792.383410] Lustre: Skipped 1 previous similar message [ 3792.397477] Lustre: Skipped 1 previous similar message [ 3796.896620] Lustre: server umount lustre-OST0000 complete [ 3801.579953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3801.597257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.621929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3807.667196] Lustre: server umount lustre-MDT0000 complete [ 3807.721824] LustreError: 109866:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.740946] LustreError: 109866:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3812.562532] LustreError: 109848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774878206 with bad export cookie 9171289163091750109 [ 3812.570491] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3812.575227] LustreError: 109848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3812.980719] Lustre: server umount lustre-MDT0001 complete [ 3827.220812] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 3830.619028] Key type lgssc unregistered [ 3830.998913] LNet: 113073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3831.014429] LNetError: 113073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3831.032960] LNet: Removed LNI 192.168.202.108@tcp [ 3832.153951] Key type .llcrypt unregistered [ 3832.157347] Key type ._llcrypt unregistered [ 3862.516343] Key type ._llcrypt registered [ 3862.517965] Key type .llcrypt registered [ 3862.707784] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 3864.674790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3864.779083] alg: No test for adler32 (adler32-zlib) [ 3865.985614] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 3866.337100] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 3868.040192] Key type lgssc registered [ 3870.065214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3879.448357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3879.463743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.155237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.432508] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3890.671797] LustreError: 114476:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.701876] LustreError: 114476:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3895.580827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.778681] LustreError: 114495:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.023485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3900.997491] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3908.097068] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3912.192865] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3919.408978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.758508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.034308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3927.319813] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 3934.904280] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3943.553491] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3948.431722] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3948.440270] Lustre: Skipped 2 previous similar messages [ 3951.781214] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3951.853437] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3953.770366] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3953.775565] Lustre: Skipped 2 previous similar messages [ 3960.946946] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3960.952928] Lustre: Skipped 1 previous similar message [ 3966.194971] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.8@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3966.200991] Lustre: Skipped 2 previous similar messages [ 3968.481748] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.8@tcp 192.168.0.[2,10]@tcp [ 3973.251943] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.8/24@tcp in log lustre-MDT0000 [ 3973.257325] Lustre: Skipped 2 previous similar messages [ 3975.765145] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.8/24@tcp [ 3975.784190] Lustre: Skipped 1 previous similar message [ 3987.425305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3987.430261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.434633] LustreError: Skipped 1 previous similar message [ 3987.436793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3987.465983] Lustre: Skipped 1 previous similar message [ 3989.437486] Lustre: server umount lustre-OST0000 complete [ 3997.672752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.686562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.689506] Lustre: Skipped 1 previous similar message [ 3997.707625] Lustre: Skipped 2 previous similar messages [ 4001.461178] Lustre: server umount lustre-MDT0000 complete [ 4002.793259] LustreError: 114471:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4002.811687] LustreError: 114471:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4005.867736] LustreError: 116093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774878399 with bad export cookie 17055650142933595579 [ 4005.874756] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4006.325632] Lustre: server umount lustre-MDT0001 complete [ 4019.903835] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 4023.698273] Key type lgssc unregistered [ 4024.001720] LNet: 118923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4024.010329] LNetError: 118923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4025.070655] LNet: Removed LNI 192.168.202.108@tcp [ 4026.100408] Key type .llcrypt unregistered [ 4026.108374] Key type ._llcrypt unregistered [ 4058.404326] Key type ._llcrypt registered [ 4058.409316] Key type .llcrypt registered [ 4058.590904] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4059.961189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4060.098540] alg: No test for adler32 (adler32-zlib) [ 4061.226785] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 4061.550295] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 4063.232467] Key type lgssc registered [ 4064.977448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4074.799285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4074.825445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.438667] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4076.462333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.482308] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.8/24@tcp [ 4082.018952] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4086.767370] LustreError: 120479:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.787792] LustreError: 120479:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4091.605228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.874743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4092.052244] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4092.079966] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.8/24@tcp [ 4092.114400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4096.606916] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4102.546381] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4108.177702] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4118.289895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4118.623458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4121.720262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4126.494085] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4133.043553] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4152.801529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4152.802372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.813824] LustreError: Skipped 1 previous similar message [ 4152.817212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4152.866701] Lustre: Skipped 1 previous similar message [ 4155.303515] Lustre: server umount lustre-OST0000 complete [ 4163.050622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4163.057950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4163.069905] Lustre: Skipped 1 previous similar message [ 4163.105217] Lustre: Skipped 2 previous similar messages [ 4165.855355] Lustre: server umount lustre-MDT0000 complete [ 4168.162225] LustreError: 120478:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.176230] LustreError: 120478:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4170.454127] LustreError: 120458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774878564 with bad export cookie 9098470600197903753 [ 4170.467794] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4170.473580] LustreError: 120458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4171.269930] Lustre: server umount lustre-MDT0001 complete [ 4184.051293] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 4187.452222] Key type lgssc unregistered [ 4187.836831] LNet: 123716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4187.844597] LNetError: 123716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4187.869803] LNet: Removed LNI 192.168.202.108@tcp [ 4189.017204] Key type .llcrypt unregistered [ 4189.020587] Key type ._llcrypt unregistered [ 4220.758333] Key type ._llcrypt registered [ 4220.762656] Key type .llcrypt registered [ 4221.005547] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 4237.641496] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4239.126935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4239.162098] alg: No test for adler32 (adler32-zlib) [ 4240.602554] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 4240.981600] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 4242.720208] Key type lgssc registered [ 4244.003323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4286.229709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4286.322337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.591236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4287.631799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4287.740974] Lustre: lustre-MDT0000: new disk, initializing [ 4287.850973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.870487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4292.629717] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4305.418180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.561608] Lustre: 127090:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4305.624434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4305.632337] Lustre: Skipped 1 previous similar message [ 4305.715874] Lustre: lustre-MDT0001: new disk, initializing [ 4305.805152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4305.841728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4305.856239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4310.974247] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4319.623387] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4323.568494] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4331.366309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4331.685938] Lustre: lustre-OST0000: new disk, initializing [ 4331.688804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4331.813946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4339.363251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4339.382983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4339.547607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4339.676423] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4347.872700] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4354.695740] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4355.042177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4362.415663] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4362.805380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4370.401728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4370.409670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.429469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4370.437291] Lustre: Skipped 1 previous similar message [ 4370.462154] Lustre: Skipped 1 previous similar message [ 4371.925241] Lustre: server umount lustre-OST0000 complete [ 4377.576302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4377.592158] LustreError: Skipped 1 previous similar message [ 4377.600274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.617688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4380.643100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.644471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.184555] Lustre: server umount lustre-MDT0000 complete [ 4387.844354] LustreError: 127082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774878781 with bad export cookie 8061946049939064515 [ 4387.845745] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4387.852949] LustreError: 127082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4388.483922] Lustre: server umount lustre-MDT0001 complete [ 4398.100573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.891332] LustreError: 130905:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.918817] LustreError: 130905:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4399.029239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.193769] LustreError: 130906:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.211586] LustreError: 130906:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4404.228227] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4408.293955] LustreError: 130905:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.412252] LustreError: 130906:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.986414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.560466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4421.171564] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4428.126850] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4433.089778] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4441.347441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4441.767258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.422943] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4456.604498] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4463.022309] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4463.348683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4469.536890] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4469.941802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4484.133174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4484.333759] Lustre: lustre-OST0001: new disk, initializing [ 4484.338327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4484.420805] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4489.884794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4489.894421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4490.111076] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4491.883351] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4502.698396] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4503.052230] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4508.839541] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4509.165204] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4514.467436] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d0c6812800.ost_server_uuid 50 [ 4516.518106] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d0c6812800.ost_server_uuid in IDLE state after 0 sec [ 4530.922182] Lustre: server umount lustre-OST0001 complete [ 4531.185330] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4531.189644] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4531.203326] LustreError: 132498:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4531.203706] Lustre: Skipped 1 previous similar message [ 4531.231486] LustreError: 132498:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 4536.294269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.295367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4536.307053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4536.307062] Lustre: Skipped 1 previous similar message [ 4536.337397] LustreError: Skipped 1 previous similar message [ 4541.411221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.420070] LustreError: 133401:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.424956] Lustre: Skipped 2 previous similar messages [ 4541.484014] LustreError: 133401:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4550.112147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4550.307910] Lustre: server umount lustre-OST0000 complete [ 4556.770478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4556.791316] Lustre: Skipped 1 previous similar message [ 4556.794647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.797379] Lustre: Skipped 2 previous similar messages [ 4561.739598] Lustre: server umount lustre-MDT0000 complete [ 4561.897651] LustreError: 130905:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.919326] LustreError: 130905:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 4566.530340] LustreError: 130887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774878960 with bad export cookie 8061946049939065397 [ 4566.536710] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.541063] LustreError: 130887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4567.418203] Lustre: server umount lustre-MDT0001 complete [ 4594.564207] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 4597.243983] Key type lgssc unregistered [ 4597.680845] LNet: 136751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4597.691906] LNetError: 136751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4597.734389] LNet: Removed LNI 192.168.202.108@tcp [ 4598.731940] Key type .llcrypt unregistered [ 4598.736406] Key type ._llcrypt unregistered [ 4643.857883] Key type ._llcrypt registered [ 4643.860613] Key type .llcrypt registered [ 4643.969911] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 4659.923388] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4662.214052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4662.284322] alg: No test for adler32 (adler32-zlib) [ 4663.627653] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 4663.890529] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 4665.600207] Key type lgssc registered [ 4667.298871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4712.426235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4712.448130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.875978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4713.950612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4714.095977] Lustre: lustre-MDT0000: new disk, initializing [ 4714.225628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4714.258437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4719.941719] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4734.022551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.271666] Lustre: 140875:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4734.365194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4734.368202] Lustre: Skipped 1 previous similar message [ 4734.450781] Lustre: lustre-MDT0001: new disk, initializing [ 4734.624680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4734.678992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4734.687174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4740.801457] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4750.243765] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4754.865505] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4763.910045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.394567] Lustre: lustre-OST0000: new disk, initializing [ 4764.401957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4764.565949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.917879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4769.933027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4770.089923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4773.145840] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4783.185853] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4788.534903] Lustre: Failing over lustre-OST0000 [ 4788.728532] Lustre: server umount lustre-OST0000 complete [ 4806.113275] Lustre: 139096:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774879183/real 1774879183] req@000000007347ddd8 x1861095582106880/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1774879199 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4806.159747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.136651] Lustre: 139098:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774879184/real 1774879184] req@000000009de6a33a x1861095582107264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1774879200 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4807.137443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.187048] Lustre: 139098:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4811.345191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.974807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.995542] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4812.244565] Lustre: 139099:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774879189/real 1774879189] req@00000000126f1d0f x1861095582108800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774879205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4812.303671] Lustre: 139099:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4816.042293] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4817.056128] Lustre: 139099:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774879194/real 1774879194] req@000000001c9f0c54 x1861095582109824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774879210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4817.082052] Lustre: 139099:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4817.229455] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4817.232731] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4820.561403] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4822.181110] Lustre: 139097:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774879199/real 1774879199] req@0000000073edb1a9 x1861095582110976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1774879215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4824.692080] Lustre: Failing over lustre-MDT0000 [ 4825.068558] Lustre: server umount lustre-MDT0000 complete [ 4826.596227] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4826.598593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4826.608144] Lustre: Skipped 1 previous similar message [ 4831.714486] LustreError: 142005:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4831.756388] LustreError: 142005:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 7 previous similar messages [ 4832.262251] LustreError: 140881:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4836.848700] LustreError: 140882:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4836.907513] LustreError: 140882:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4841.970842] LustreError: 140885:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.006667] LustreError: 140885:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 4842.976549] Lustre: 139098:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774879220/real 1774879220] req@00000000de449b83 x1861095582123264/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774879236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4843.030583] Lustre: 139098:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4843.038599] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4845.969550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.621634] LustreError: 140882:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.651614] LustreError: 140882:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 4853.230989] LustreError: 139095:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003dc48201 x1861095582129280/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4853.618866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.687654] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4857.856347] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4858.855529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4858.864486] Lustre: Skipped 1 previous similar message [ 4858.901300] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4858.970063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4859.327980] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4873.698357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4873.710422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4873.734291] Lustre: Skipped 1 previous similar message [ 4873.747683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4876.355740] Lustre: server umount lustre-OST0000 complete [ 4884.450431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.450607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4884.479961] Lustre: Skipped 1 previous similar message [ 4884.482235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4884.541227] Lustre: Skipped 3 previous similar messages [ 4887.580962] Lustre: server umount lustre-MDT0000 complete [ 4889.577733] LustreError: 142116:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.630120] LustreError: 142116:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 4892.784990] LustreError: 140866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774879286 with bad export cookie 18181863294514823273 [ 4892.794649] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4892.801574] LustreError: 140866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4893.285517] Lustre: server umount lustre-MDT0001 complete [ 4908.191378] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 4912.386478] Key type lgssc unregistered [ 4912.833071] LNet: 146134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4912.836834] LNetError: 146134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4913.902853] LNet: Removed LNI 192.168.202.108@tcp [ 4914.914276] Key type .llcrypt unregistered [ 4914.916590] Key type ._llcrypt unregistered [ 4959.599768] Key type ._llcrypt registered [ 4959.617432] Key type .llcrypt registered [ 4959.815403] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 4961.062154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4961.365605] alg: No test for adler32 (adler32-zlib) [ 4962.662968] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 4963.041634] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 4964.755610] Key type lgssc registered [ 4966.163087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4974.909977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4974.964890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.691973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.808563] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 4986.851793] LustreError: 147642:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4986.885706] LustreError: 147642:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 4991.972435] LustreError: 147641:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4993.714940] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4997.094026] LustreError: 147642:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4998.782631] LustreError: 147671:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4998.794658] LustreError: 147671:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4998.802356] LustreError: 147671:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4999.111540] Lustre: server umount lustre-MDT0000 complete [ 5019.366952] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 5031.383959] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5076.595755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.756647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5076.786275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5076.840818] Lustre: lustre-MDT0000: new disk, initializing [ 5076.887551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5076.900083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5083.395355] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5095.629613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.746850] Lustre: 150830:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5095.790515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5095.793734] Lustre: Skipped 1 previous similar message [ 5095.849276] Lustre: lustre-MDT0001: new disk, initializing [ 5095.916257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5095.947653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5095.957511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5101.361820] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5111.299882] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5117.787991] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5127.377461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5127.782348] Lustre: lustre-OST0000: new disk, initializing [ 5127.787967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5127.922249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.373889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5134.382386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5134.459666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5134.611732] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5144.054669] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5148.758515] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5150.705186] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:05:41 (1774879541) [ 5438.950507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5438.957045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.989540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5441.506266] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.510224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5441.533333] Lustre: Skipped 1 previous similar message [ 5446.625587] LustreError: 150835:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.658279] LustreError: 150835:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5447.047873] Lustre: server umount lustre-MDT0000 complete [ 5450.547231] LustreError: 150821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774879844 with bad export cookie 12586585279553588376 [ 5450.551456] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5450.552740] LustreError: 150821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5451.744843] LustreError: 154228:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.744983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.766665] Lustre: Skipped 1 previous similar message [ 5451.769749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5451.772427] Lustre: Skipped 1 previous similar message [ 5455.778902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5456.876438] Lustre: server umount lustre-MDT0001 complete [ 5467.703245] Lustre: server umount lustre-OST0000 complete [ 5478.288655] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 5489.709632] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5534.303764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.814298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5534.874748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5535.030570] Lustre: lustre-MDT0000: new disk, initializing [ 5535.309195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.357496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5542.347429] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5554.958544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.148568] Lustre: 157099:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5555.254552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5555.260888] Lustre: Skipped 1 previous similar message [ 5555.405192] Lustre: lustre-MDT0001: new disk, initializing [ 5555.556733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5555.626343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5555.648268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5560.818193] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5569.039828] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5572.987449] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5581.147930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5581.418338] Lustre: lustre-OST0000: new disk, initializing [ 5581.423688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5581.551485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5589.136701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5589.154977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5589.334952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5589.812621] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5598.768621] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5604.283255] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5604.566606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5610.161981] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5610.465933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5615.076563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.077541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5615.110283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5615.123292] Lustre: Skipped 1 previous similar message [ 5625.314069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5625.317972] Lustre: Skipped 3 previous similar messages [ 5627.360158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5627.759730] Lustre: server umount lustre-OST0000 complete [ 5635.555297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.573929] Lustre: Skipped 1 previous similar message [ 5639.217715] Lustre: server umount lustre-MDT0000 complete [ 5640.673220] LustreError: 160036:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.702809] LustreError: 160036:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 5643.435227] LustreError: 157093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774880037 with bad export cookie 12586585279553747206 [ 5643.452620] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5643.454635] LustreError: 157093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5644.124629] Lustre: server umount lustre-MDT0001 complete [ 5653.518719] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:14:04 (1774880044) [ 5677.207549] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 5690.601381] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5737.581325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.065704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5738.107265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5738.194815] Lustre: lustre-MDT0000: new disk, initializing [ 5738.335912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.377211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5743.043490] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5755.164531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.276446] Lustre: 163425:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5755.560434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5760.728680] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5769.742796] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5774.018539] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5783.579491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5784.055297] Lustre: lustre-OST0000: new disk, initializing [ 5784.057402] Lustre: Skipped 1 previous similar message [ 5784.061559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5784.073292] Lustre: Skipped 2 previous similar messages [ 5790.901683] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5792.943399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5792.953232] Lustre: Skipped 1 previous similar message [ 5792.968973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5793.139131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5801.542485] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5818.853130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.873561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5818.878368] Lustre: Skipped 2 previous similar messages [ 5818.885177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5818.885187] Lustre: Skipped 2 previous similar messages [ 5820.374281] Lustre: server umount lustre-OST0000 complete [ 5827.553364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5831.698203] Lustre: server umount lustre-MDT0000 complete [ 5834.222821] LustreError: 163432:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.251875] LustreError: 163432:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 5836.675919] LustreError: 163417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774880230 with bad export cookie 12586585279553748088 [ 5836.681514] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.689648] LustreError: 163417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5837.273239] Lustre: server umount lustre-MDT0001 complete [ 5847.272127] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:17:18 (1774880238) [ 5869.883211] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 5881.523560] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 5927.685862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.153741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5928.261804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5928.376273] Lustre: lustre-MDT0000: new disk, initializing [ 5928.504941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.510695] Lustre: Skipped 2 previous similar messages [ 5928.532876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5934.014751] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5946.104695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.302178] Lustre: 169356:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5946.730545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5952.171580] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5960.718208] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5964.729288] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5973.027433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5980.327724] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 5982.858292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5983.040131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5988.776117] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5998.562916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5998.566103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.577440] LustreError: Skipped 1 previous similar message [ 5998.612500] Lustre: Skipped 2 previous similar messages [ 5998.620510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5998.620520] Lustre: Skipped 4 previous similar messages [ 6003.444462] Lustre: server umount lustre-OST0000 complete [ 6008.290200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6013.923092] LustreError: 170548:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.978212] LustreError: 170548:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6014.078247] Lustre: server umount lustre-MDT0000 complete [ 6018.868974] LustreError: 169348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774880412 with bad export cookie 12586585279553749173 [ 6018.879351] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6018.889635] LustreError: 169348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6019.507870] Lustre: server umount lustre-MDT0001 complete [ 6036.263851] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6040.179559] Key type lgssc unregistered [ 6040.628186] LNet: 172835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6040.641738] LNetError: 172835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6040.657841] LNet: Removed LNI 192.168.202.108@tcp [ 6041.116643] Key type .llcrypt unregistered [ 6041.117541] Key type ._llcrypt unregistered [ 6068.222267] Key type ._llcrypt registered [ 6068.226367] Key type .llcrypt registered [ 6068.292672] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6068.973713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6068.996798] alg: No test for adler32 (adler32-zlib) [ 6070.248864] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 6070.535390] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 6072.168371] Key type lgssc registered [ 6073.376986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6083.532866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6083.598866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.629098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6092.413562] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6095.842373] LustreError: 174239:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6095.853762] LustreError: 174239:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6100.968890] LustreError: 174240:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.818944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.357256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6109.438892] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6116.769986] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6121.495316] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6129.225168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6129.603101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.759450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6136.287675] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6144.527472] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6166.499408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6166.508880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6166.511474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6166.526508] Lustre: Skipped 1 previous similar message [ 6166.577598] LustreError: Skipped 1 previous similar message [ 6170.144770] Lustre: server umount lustre-OST0000 complete [ 6176.737508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6176.754952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6176.765858] Lustre: Skipped 2 previous similar messages [ 6181.167446] Lustre: server umount lustre-MDT0000 complete [ 6181.860853] LustreError: 174239:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.885797] LustreError: 174239:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6186.621355] LustreError: 174220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774880580 with bad export cookie 3861367260067859239 [ 6186.645761] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6187.272920] Lustre: server umount lustre-MDT0001 complete [ 6200.589923] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 6203.969564] Key type lgssc unregistered [ 6204.374352] LNet: 177447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6204.394798] LNetError: 177447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6204.418438] LNet: Removed LNI 192.168.202.108@tcp [ 6205.836553] Key type .llcrypt unregistered [ 6205.838874] Key type ._llcrypt unregistered [ 6236.920098] Key type ._llcrypt registered [ 6236.925819] Key type .llcrypt registered [ 6237.086361] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6238.623246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6238.662245] alg: No test for adler32 (adler32-zlib) [ 6239.786516] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 6240.048218] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 6241.768400] Key type lgssc registered [ 6242.870723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6252.553963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6252.592542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.260278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6259.821872] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6263.780741] LustreError: 178854:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.799843] LustreError: 178854:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 6268.897623] LustreError: 178855:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.309620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6271.722848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6276.689638] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6282.887560] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6287.620275] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6296.421708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6296.900264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6298.938364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6304.605523] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6311.230506] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6319.593254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.597115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6319.611480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.633850] Lustre: Skipped 1 previous similar message [ 6324.730208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6324.746099] Lustre: Skipped 2 previous similar messages [ 6325.243959] Lustre: server umount lustre-OST0000 complete [ 6331.483486] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89d0c7097800.ost_server_uuid 50 [ 6333.290702] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6348.782738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6348.796173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.809971] Lustre: Skipped 1 previous similar message [ 6348.817413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6348.822909] Lustre: Skipped 1 previous similar message [ 6349.873261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.116990] Lustre: server umount lustre-MDT0000 complete [ 6354.915089] LustreError: 179565:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.934171] LustreError: 179565:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6358.641991] LustreError: 178836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774880752 with bad export cookie 14777650473327782386 [ 6358.658771] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6358.662831] LustreError: 178836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6359.552213] Lustre: server umount lustre-MDT0001 complete [ 6370.309451] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:26:01 (1774880761) [ 6378.398533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6379.026377] LustreError: 182338:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6379.117907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.271487] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6384.614486] LustreError: 182339:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.831392] LustreError: 182338:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.849291] LustreError: 182338:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 6393.961041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.406482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6399.594220] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6407.344501] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6413.674292] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6422.460211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6422.841131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.872078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6431.088529] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6437.376989] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6445.953952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6446.159509] Lustre: lustre-OST0001: new disk, initializing [ 6446.166130] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6446.319356] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6453.957635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6453.999216] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6454.224356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6454.298430] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6466.103520] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6469.607494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6469.610959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6469.630862] Lustre: Skipped 1 previous similar message [ 6469.633227] Lustre: Skipped 2 previous similar messages [ 6474.727386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6474.744969] Lustre: Skipped 3 previous similar messages [ 6475.432165] Lustre: server umount lustre-OST0000 complete [ 6477.314866] LustreError: 183932:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.205094] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6483.530771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6488.022094] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6488.217423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6505.286980] Lustre: server umount lustre-OST0001 complete [ 6510.561322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.565040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6510.577627] Lustre: Skipped 1 previous similar message [ 6510.588210] Lustre: Skipped 1 previous similar message [ 6518.863817] Lustre: server umount lustre-MDT0000 complete [ 6520.812477] LustreError: 182339:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.856261] LustreError: 182339:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 15 previous similar messages [ 6521.696104] Lustre: 178165:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774880899/real 1774880899] req@0000000070bb05c7 x1861097234729856/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1774880915 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6521.734644] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.688047] LustreError: 182319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774880917 with bad export cookie 14777650473327783457 [ 6523.690207] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.701171] LustreError: 182319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6524.181706] Lustre: server umount lustre-MDT0001 complete [ 6550.976948] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:29:02 (1774880942) [ 6558.343857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6558.731567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6558.776396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6559.008463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6564.443491] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6573.006438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6573.209746] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6573.266395] Lustre: 188141:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6579.544285] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6586.429862] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6590.585947] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6597.971732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6598.206614] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6605.227584] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6608.064663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6611.555235] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6619.664132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6619.768253] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6627.991685] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6634.417403] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6638.562104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6638.577242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6638.648229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6638.666096] Lustre: Skipped 2 previous similar messages [ 6643.415607] Lustre: server umount lustre-OST0000 complete [ 6644.715283] LustreError: 189763:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.771120] LustreError: 189763:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 6662.690417] Lustre: server umount lustre-OST0001 complete [ 6675.426310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6675.434232] Lustre: Skipped 6 previous similar messages [ 6677.700759] Lustre: server umount lustre-MDT0000 complete [ 6680.544490] Lustre: 178166:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774881058/real 1774881058] req@0000000036d2f675 x1861097234794240/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1774881074 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6680.573525] Lustre: 178166:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6680.589230] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.605994] Lustre: Skipped 3 previous similar messages [ 6682.299724] LustreError: 190688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774881076 with bad export cookie 14777650473327784864 [ 6682.303388] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6682.311785] LustreError: 190688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6682.708674] Lustre: server umount lustre-MDT0001 complete [ 6710.452962] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:31:41 (1774881101) [ 6731.475235] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 6741.555463] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 6782.102468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.396354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6782.432783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6782.536652] Lustre: lustre-MDT0000: new disk, initializing [ 6782.625992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6782.628702] Lustre: Skipped 3 previous similar messages [ 6782.645222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6787.343528] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6798.564722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6798.702703] Lustre: 195753:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6798.785624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6798.789586] Lustre: Skipped 1 previous similar message [ 6798.899387] Lustre: lustre-MDT0001: new disk, initializing [ 6799.120227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6799.140733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6803.644803] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6811.570703] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6815.666257] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6822.662421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6822.913927] Lustre: lustre-OST0000: new disk, initializing [ 6822.922624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6830.372439] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6832.752296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6832.757286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6832.905070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6840.460989] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6845.784666] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6846.057511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6851.553885] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6851.897026] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6858.721203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6858.744523] LustreError: Skipped 1 previous similar message [ 6858.765917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.814453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6860.860428] Lustre: server umount lustre-OST0000 complete [ 6865.892949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6865.900634] LustreError: Skipped 1 previous similar message [ 6871.870499] Lustre: server umount lustre-MDT0000 complete [ 6874.086872] LustreError: 196811:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6874.106569] LustreError: 196811:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 12 previous similar messages [ 6876.193281] LustreError: 195746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774881269 with bad export cookie 14777650473327786425 [ 6876.199362] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6876.211313] LustreError: 195746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6876.620436] Lustre: server umount lustre-MDT0001 complete [ 6886.001463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.729429] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6907.549852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.455639] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6920.579850] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6926.214628] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6934.997170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.505946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6935.510302] Lustre: Skipped 4 previous similar messages [ 6944.134575] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 6950.849437] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6955.778631] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6958.123177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6963.096833] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6963.441819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6967.271841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.440706] Lustre: server umount lustre-OST0000 complete [ 6979.570706] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6985.525981] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7015.257845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7022.896255] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7029.776629] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7032.698820] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 7033.256444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7033.266137] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7033.276326] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7033.281843] Lustre: Skipped 1 previous similar message [ 7038.434722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.451852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7038.452590] Lustre: Skipped 5 previous similar messages [ 7038.458719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7038.480051] Lustre: Skipped 7 previous similar messages [ 7043.838520] Lustre: server umount lustre-OST0000 complete [ 7056.150313] Lustre: server umount lustre-MDT0000 complete [ 7060.831550] LustreError: 199547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774881454 with bad export cookie 14777650473327787307 [ 7060.860182] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7060.865548] LustreError: 199547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7061.593882] Lustre: server umount lustre-MDT0001 complete [ 7073.847451] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:37:45 (1774881465) [ 7081.259332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.640541] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7097.574147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.599761] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7110.661582] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7115.868445] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7123.975949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7134.200545] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7143.743403] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7150.019699] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7150.316409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7155.392688] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7155.627476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7163.112557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7163.300820] Lustre: lustre-OST0001: new disk, initializing [ 7163.306892] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7171.249852] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7173.299172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7173.307943] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7173.402451] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7179.741276] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7185.700222] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7185.966976] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7191.795454] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7192.135833] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7199.209081] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7199.234266] LustreError: Skipped 2 previous similar messages [ 7200.808607] Lustre: server umount lustre-OST0001 complete [ 7204.345779] LustreError: 206634:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.382855] LustreError: 206634:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 12 previous similar messages [ 7210.037210] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7210.308318] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7216.937650] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7217.295451] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7249.512067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.898143] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7249.901452] Lustre: Skipped 5 previous similar messages [ 7255.014997] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7255.030528] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7255.054444] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7258.997246] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7266.418622] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7268.935909] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7275.491118] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7275.502108] LustreError: Skipped 1 previous similar message [ 7277.733899] Lustre: server umount lustre-OST0001 complete [ 7295.969608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.980556] Lustre: Skipped 7 previous similar messages [ 7296.001889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7296.018584] Lustre: Skipped 9 previous similar messages [ 7301.968634] Lustre: server umount lustre-OST0000 complete [ 7312.489858] Lustre: server umount lustre-MDT0000 complete [ 7316.955637] LustreError: 205018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774881710 with bad export cookie 14777650473327788273 [ 7316.965950] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7316.967372] LustreError: 205018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7331.296308] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7331.785296] Lustre: server umount lustre-MDT0001 complete [ 7361.614863] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:42:32 (1774881752) [ 7368.916378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.237418] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7369.270762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7374.764820] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7385.418612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.558211] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7385.575975] Lustre: 213006:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7391.201483] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7398.195118] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7403.244543] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7410.920983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.269417] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7419.222722] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7425.351760] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7435.267266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.402345] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7442.626586] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7449.604178] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7451.828512] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7457.337745] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7457.686420] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7463.295613] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7463.612933] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7468.361654] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d0d975f800.ost_server_uuid 50 [ 7470.237589] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d0d975f800.ost_server_uuid in IDLE state after 0 sec [ 7471.608395] Lustre: Permanently deactivating lustre-OST0001 [ 7471.624559] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7476.016164] Lustre: Permanently reactivating lustre-OST0001 [ 7476.038351] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7476.041754] Lustre: Skipped 2 previous similar messages [ 7481.832972] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7481.848201] LustreError: Skipped 1 previous similar message [ 7487.207464] Lustre: server umount lustre-OST0001 complete [ 7497.803238] Lustre: server umount lustre-OST0000 complete [ 7507.930038] Lustre: server umount lustre-MDT0000 complete [ 7511.516462] LustreError: 212996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774881905 with bad export cookie 14777650473327789561 [ 7511.524737] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7511.528768] LustreError: 212996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7511.827844] Lustre: server umount lustre-MDT0001 complete [ 7537.775238] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:45:29 (1774881929) [ 7546.900546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7547.327498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7547.357991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7547.372303] Lustre: Skipped 2 previous similar messages [ 7552.357922] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7561.900622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.046750] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7562.065425] Lustre: 219120:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7567.314572] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7574.200433] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7578.416279] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7586.459606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7586.730537] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7586.769224] Lustre: Permanently deactivating lustre-OST0000 [ 7586.779464] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7593.733977] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7595.539641] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7599.901302] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7607.591635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7607.711961] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7614.134901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7614.394425] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7621.141954] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7623.473218] Lustre: Permanently reactivating lustre-OST0000 [ 7623.516654] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7623.519715] Lustre: Skipped 2 previous similar messages [ 7629.901529] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7629.916604] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 34 seconds. I think it's dead, and I am evicting it. exp 0000000008516c51, cur 1774882023 deadline 1774882019 last 1774881989 [ 7630.006296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7642.779911] Lustre: server umount lustre-OST0001 complete [ 7653.049657] Lustre: server umount lustre-OST0000 complete [ 7672.288214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7672.463592] Lustre: server umount lustre-MDT0000 complete [ 7676.498635] LustreError: 220760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774882070 with bad export cookie 14777650473327791234 [ 7676.499995] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.513576] LustreError: 220760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7676.959292] Lustre: server umount lustre-MDT0001 complete [ 7687.894447] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 7690.655322] Key type lgssc unregistered [ 7690.899176] LNet: 223521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7690.910593] LNetError: 223521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7690.934994] LNet: Removed LNI 192.168.202.108@tcp [ 7691.949171] Key type .llcrypt unregistered [ 7691.951155] Key type ._llcrypt unregistered [ 7713.308300] Key type ._llcrypt registered [ 7713.310579] Key type .llcrypt registered [ 7713.401874] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 7714.718391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7714.766132] alg: No test for adler32 (adler32-zlib) [ 7715.785186] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 7716.104642] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 7717.776561] Key type lgssc registered [ 7718.950731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7730.899994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7730.916873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.473823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.090107] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7742.952726] LustreError: 224963:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.986920] LustreError: 224963:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 7743.971647] LustreError: 224964:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.160945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7748.361847] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7748.410610] Lustre: 224951:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7748.422621] Lustre: Permanently deactivating lustre-MDT0001 [ 7748.439633] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7748.664318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7752.965621] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7758.663633] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7758.870585] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7762.957355] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7772.054277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7772.476244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7773.504423] LustreError: 226577:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7778.666877] LustreError: 226579:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7778.682967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7778.689708] LustreError: 226579:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7781.199977] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7783.907127] LustreError: 226577:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7783.935970] LustreError: 226577:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 7788.164679] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7794.160047] LustreError: 226577:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.185143] LustreError: 226577:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7796.499146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7796.902757] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7802.370568] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7802.393071] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7804.458353] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7812.074848] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7815.706904] Lustre: Permanently reactivating lustre-MDT0001 [ 7815.751647] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7815.754531] Lustre: Skipped 2 previous similar messages [ 7822.456365] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.476318] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7822.486799] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7822.498292] LustreError: 228088:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7822.500607] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7822.507917] LustreError: 228088:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7822.508076] LustreError: 228088:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.108@tcp: cfg command failed: rc = -108 [ 7822.508806] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7822.508806] [ 7822.509576] LustreError: 224955:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7822.888189] Lustre: lustre-MDT0001: Client 396a2319-75a7-4841-bc3e-b891c78d74b9 (at 192.168.202.8@tcp) reconnecting [ 7827.095883] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7829.322200] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7835.950887] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7836.170986] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7841.652617] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7841.970487] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7846.060473] Lustre: Permanently deactivating lustre-MDT0001 [ 7846.108837] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7846.116476] Lustre: Skipped 2 previous similar messages [ 7856.387985] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7866.337336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7866.340611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.349514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.903771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.910184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.922149] Lustre: Skipped 2 previous similar messages [ 7868.932609] Lustre: Skipped 3 previous similar messages [ 7871.213221] Lustre: server umount lustre-MDT0000 complete [ 7874.032928] LustreError: 224963:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.067957] LustreError: 224963:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 7875.845052] LustreError: 228068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774882269 with bad export cookie 15186562401343409109 [ 7875.846745] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7875.853540] LustreError: 228068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7876.324206] Lustre: server umount lustre-MDT0001 complete [ 7883.242770] Lustre: server umount lustre-OST0000 complete [ 7896.255742] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:51:27 (1774882287) [ 7908.754738] Lustre: server umount lustre-OST0001 complete [ 7915.583127] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 7926.094271] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 7967.225661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.587206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7967.601387] Lustre: Skipped 2 previous similar messages [ 7967.674276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7967.865291] Lustre: lustre-MDT0000: new disk, initializing [ 7967.982457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7967.993883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7974.079754] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 7988.334892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.486117] Lustre: 232667:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7988.547475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7988.556017] Lustre: Skipped 1 previous similar message [ 7988.687675] Lustre: lustre-MDT0001: new disk, initializing [ 7988.744524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7988.765035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7988.770109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7993.390266] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8002.603518] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8006.630891] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8015.066083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8015.489750] Lustre: lustre-OST0000: new disk, initializing [ 8015.502928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8015.649379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8022.574625] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8025.718214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8025.734639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8025.877902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8030.693786] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8033.229129] LustreError: 235014:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8043.118316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8043.249100] LustreError: 235014:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8043.293358] LustreError: 232674:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8043.344774] Lustre: lustre-OST0001: new disk, initializing [ 8043.347579] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8049.843947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8049.851731] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8050.007978] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8051.116589] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8053.344404] LustreError: 232674:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8059.283990] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8060.759608] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8065.506638] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8065.514357] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.517992] LustreError: Skipped 1 previous similar message [ 8065.518853] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8065.563201] Lustre: Skipped 1 previous similar message [ 8069.406562] Lustre: server umount lustre-OST0001 complete [ 8070.628980] LustreError: 234392:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.653383] LustreError: 234392:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8087.527708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8087.530913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.538683] LustreError: Skipped 1 previous similar message [ 8087.542201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8087.552553] Lustre: Skipped 1 previous similar message [ 8087.567213] Lustre: Skipped 2 previous similar messages [ 8093.902678] Lustre: server umount lustre-OST0000 complete [ 8101.348794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.352269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8101.353746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.353754] Lustre: Skipped 2 previous similar messages [ 8101.364153] Lustre: Skipped 1 previous similar message [ 8104.587727] Lustre: server umount lustre-MDT0000 complete [ 8108.669787] LustreError: 232658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774882502 with bad export cookie 15186562401343411706 [ 8108.676286] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8108.679799] LustreError: 232658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8115.405099] Lustre: server umount lustre-MDT0001 complete [ 8127.913544] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 8130.721803] Key type lgssc unregistered [ 8130.988706] LNet: 237317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8130.993085] LNetError: 237317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8131.009588] LNet: Removed LNI 192.168.202.108@tcp [ 8131.857083] Key type .llcrypt unregistered [ 8131.859380] Key type ._llcrypt unregistered [ 8172.793846] Key type ._llcrypt registered [ 8172.796629] Key type .llcrypt registered [ 8172.932918] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 8174.400687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8174.445900] alg: No test for adler32 (adler32-zlib) [ 8175.488747] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 8175.766787] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 8177.515057] Key type lgssc registered [ 8178.775268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8187.140347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8187.164649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8188.497384] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8188.518345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8188.688269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8193.078334] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8202.271159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8202.379905] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8202.411681] Lustre: 239489:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8202.700723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8208.085971] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8213.213737] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8217.074146] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8225.067335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8225.226510] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8225.453561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8231.787678] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8233.609038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8237.732319] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8289.764606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8289.779243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.817913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8290.273572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8290.285635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.583146] Lustre: server umount lustre-OST0000 complete [ 8300.187042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8313.936541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.261490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8320.458593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8320.476211] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8321.219530] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8325.624531] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8325.653342] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8328.011181] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8371.681198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8371.694466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8371.696021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8371.711749] Lustre: Skipped 1 previous similar message [ 8371.733892] Lustre: Skipped 3 previous similar messages [ 8374.404660] Lustre: server umount lustre-OST0000 complete [ 8381.925904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8381.931914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8381.942784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8381.947554] Lustre: Skipped 1 previous similar message [ 8381.970548] LustreError: Skipped 1 previous similar message [ 8384.585611] Lustre: server umount lustre-MDT0000 complete [ 8387.061913] LustreError: 240531:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8387.074377] LustreError: 240531:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 4 previous similar messages [ 8388.307313] LustreError: 239480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774882782 with bad export cookie 1728648182006003786 [ 8388.308307] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8388.323786] LustreError: 239480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8388.793637] Lustre: server umount lustre-MDT0001 complete [ 8399.919809] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 8402.715577] Key type lgssc unregistered [ 8403.076521] LNet: 244444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8403.081513] LNetError: 244444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8404.136558] LNet: Removed LNI 192.168.202.108@tcp [ 8405.035194] Key type .llcrypt unregistered [ 8405.037138] Key type ._llcrypt unregistered [ 8430.899673] Key type ._llcrypt registered [ 8430.902516] Key type .llcrypt registered [ 8431.062846] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 8432.574635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8432.768483] alg: No test for adler32 (adler32-zlib) [ 8433.899779] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 8434.132221] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 8435.776240] Key type lgssc registered [ 8437.130768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8445.443513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8445.465742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8447.030150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.798697] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8457.193323] LustreError: 245844:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8457.225862] LustreError: 245844:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8458.213298] LustreError: 245845:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8460.397334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8460.923263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8465.215439] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8470.568773] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8473.572817] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8479.640941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8479.930402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.718722] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8491.759702] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8493.236472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8517.089602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8517.099414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.110985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8519.139582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8519.151261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.152478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.404904] Lustre: server umount lustre-OST0000 complete [ 8527.329881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8527.346683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8527.362227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8527.375133] Lustre: Skipped 1 previous similar message [ 8529.377663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8529.383757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8530.820757] Lustre: server umount lustre-MDT0000 complete [ 8534.452794] LustreError: 245824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774882928 with bad export cookie 2834999319539363248 [ 8534.459717] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8534.464983] LustreError: 245824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8534.804926] Lustre: server umount lustre-MDT0001 complete [ 8546.068816] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 8548.931775] Key type lgssc unregistered [ 8549.230139] LNet: 249808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8549.236991] LNetError: 249808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8549.271993] LNet: Removed LNI 192.168.202.108@tcp [ 8550.393861] Key type .llcrypt unregistered [ 8550.396336] Key type ._llcrypt unregistered [ 8569.981748] Key type ._llcrypt registered [ 8569.983603] Key type .llcrypt registered [ 8570.057327] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 8571.045110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8571.165409] alg: No test for adler32 (adler32-zlib) [ 8572.118039] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 8572.535454] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 8574.456961] Key type lgssc registered [ 8576.121730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8588.397386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8588.440636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.102286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8596.893427] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8599.550935] LustreError: 251014:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.625148] LustreError: 251014:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8604.642423] LustreError: 251015:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8608.739619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8609.259704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8614.549784] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8622.024106] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8627.592226] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8636.425893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8636.953861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8645.377889] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8652.305660] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8659.453320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8678.883080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8678.892196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8678.915321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8679.910065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8679.913556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8679.934829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.026572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8685.038052] Lustre: Skipped 2 previous similar messages [ 8689.121041] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8689.397746] Lustre: server umount lustre-OST0000 complete [ 8695.267143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8695.286253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8695.294979] Lustre: Skipped 1 previous similar message [ 8695.313313] Lustre: Skipped 1 previous similar message [ 8700.385649] LustreError: 252111:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.402652] Lustre: server umount lustre-MDT0000 complete [ 8700.410539] LustreError: 252111:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8704.852817] LustreError: 250994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774883098 with bad export cookie 13652077124959937233 [ 8704.853512] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8704.878022] LustreError: 250994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8705.707517] Lustre: server umount lustre-MDT0001 complete [ 8717.441872] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 8720.607442] Key type lgssc unregistered [ 8720.889973] LNet: 254366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8720.892814] LNetError: 254366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8720.903987] LNet: Removed LNI 192.168.202.108@tcp [ 8722.137087] Key type .llcrypt unregistered [ 8722.139358] Key type ._llcrypt unregistered [ 8752.363024] Key type ._llcrypt registered [ 8752.365058] Key type .llcrypt registered [ 8752.553859] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 8754.725247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8755.051122] alg: No test for adler32 (adler32-zlib) [ 8756.374085] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 8756.718664] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 8758.531106] Key type lgssc registered [ 8760.271258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8770.188755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8770.215566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.097274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.118715] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8781.296716] LustreError: 255767:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8781.328878] LustreError: 255767:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8786.403660] LustreError: 255768:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.005595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8787.486669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8792.814973] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8799.986867] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8804.784048] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8813.200819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8813.460867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8816.574284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8821.918740] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8829.988813] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8862.690160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8862.713747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8862.743044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8863.716830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8863.730643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8863.732650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8863.756427] Lustre: Skipped 1 previous similar message [ 8866.614056] Lustre: server umount lustre-OST0000 complete [ 8873.958108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8873.963727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8873.990668] Lustre: Skipped 1 previous similar message [ 8874.009878] Lustre: Skipped 1 previous similar message [ 8877.887375] Lustre: server umount lustre-MDT0000 complete [ 8879.087181] LustreError: 255768:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8879.127986] LustreError: 255768:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 8882.530397] LustreError: 255748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774883276 with bad export cookie 12325732183088932779 [ 8882.540212] LustreError: 255748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8882.541929] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.078632] Lustre: server umount lustre-MDT0001 complete [ 8896.438987] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 8899.454809] Key type lgssc unregistered [ 8899.772740] LNet: 259724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8899.778119] LNetError: 259724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8900.839381] LNet: Removed LNI 192.168.202.108@tcp [ 8901.760630] Key type .llcrypt unregistered [ 8901.762312] Key type ._llcrypt unregistered [ 8921.480203] Key type ._llcrypt registered [ 8921.482754] Key type .llcrypt registered [ 8921.687694] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 8923.063220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8923.166109] alg: No test for adler32 (adler32-zlib) [ 8924.368216] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 8924.673097] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 8926.360303] Key type lgssc registered [ 8927.624184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8935.420372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8935.434964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.997471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8941.542850] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8947.173899] LustreError: 260944:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8947.187716] LustreError: 260944:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 8951.010161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.457909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8955.865117] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8961.696629] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8966.441676] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8973.871024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8974.407362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8979.833796] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 8981.579353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8987.070414] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9001.955965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9001.958320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.967440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9001.970428] LustreError: Skipped 1 previous similar message [ 9002.020841] Lustre: Skipped 1 previous similar message [ 9007.073312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9007.085609] Lustre: Skipped 2 previous similar messages [ 9012.195077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9016.288149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9016.471434] Lustre: server umount lustre-OST0000 complete [ 9027.559297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.578858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.587570] Lustre: Skipped 2 previous similar messages [ 9030.055568] Lustre: server umount lustre-MDT0000 complete [ 9032.686259] LustreError: 260951:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.704987] LustreError: 260951:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9034.257846] LustreError: 260909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774883427 with bad export cookie 14252626785729691162 [ 9034.265788] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9034.288636] LustreError: 260909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9034.687136] Lustre: server umount lustre-MDT0001 complete [ 9047.779995] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 9051.282337] Key type lgssc unregistered [ 9051.645972] LNet: 264295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9051.654666] LNetError: 264295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9051.684765] LNet: Removed LNI 192.168.202.108@tcp [ 9052.680740] Key type .llcrypt unregistered [ 9052.683118] Key type ._llcrypt unregistered [ 9066.746366] Key type ._llcrypt registered [ 9066.747582] Key type .llcrypt registered [ 9066.891066] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9087.391892] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 9101.728536] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 9102.887314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9103.144371] alg: No test for adler32 (adler32-zlib) [ 9104.323497] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 9104.731522] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 9106.443093] Key type lgssc registered [ 9107.829754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9149.141837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9149.167591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.559229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9150.589205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9150.718481] Lustre: lustre-MDT0000: new disk, initializing [ 9150.813348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9150.851539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9156.213864] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9168.756944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.901553] Lustre: 267802:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9168.955894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9168.959492] Lustre: Skipped 1 previous similar message [ 9169.025416] Lustre: lustre-MDT0001: new disk, initializing [ 9169.169555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9169.237053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9169.256383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9173.739509] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9181.911285] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9186.036657] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9194.472029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9194.888662] Lustre: lustre-OST0000: new disk, initializing [ 9194.894252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9195.065879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9202.121981] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9203.830186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9203.844412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9203.908614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9210.659669] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9216.481642] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9216.773576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9221.816216] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9222.034834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9224.709638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9224.719548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9224.770093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9224.785494] Lustre: Skipped 1 previous similar message [ 9229.794891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9229.810899] Lustre: Skipped 1 previous similar message [ 9230.641480] Lustre: server umount lustre-OST0000 complete [ 9235.937816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9235.961660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9235.990224] Lustre: Skipped 1 previous similar message [ 9235.993218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9240.042179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.043206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9240.077430] Lustre: Skipped 1 previous similar message [ 9242.509264] Lustre: server umount lustre-MDT0000 complete [ 9247.689526] LustreError: 268588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774883641 with bad export cookie 16637796982258902214 [ 9247.692966] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9247.703564] LustreError: 268588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9248.358242] Lustre: server umount lustre-MDT0001 complete [ 9258.898287] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:14:09 (1774883649) [ 9266.194256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9266.685170] LustreError: 271806:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9266.703327] LustreError: 271806:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 9266.795502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9271.602528] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9272.317802] LustreError: 271807:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9276.453503] LustreError: 271806:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9280.619660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.124909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9286.263711] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9291.737953] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9295.486987] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9302.591650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9302.892941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9309.862552] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9316.165234] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9327.591956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9327.594577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9327.596616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9327.642336] Lustre: Skipped 1 previous similar message [ 9333.308248] Lustre: server umount lustre-OST0000 complete [ 9337.828554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9337.863289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9337.866078] Lustre: Skipped 3 previous similar messages [ 9342.438871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9352.160179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9352.334456] Lustre: server umount lustre-MDT0000 complete [ 9353.185351] LustreError: 273117:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.207061] LustreError: 273117:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 9356.574397] LustreError: 271788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774883750 with bad export cookie 16637796982258903096 [ 9356.589274] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9356.597569] LustreError: 271788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9357.068322] Lustre: server umount lustre-MDT0001 complete [ 9370.529784] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 9374.268447] Key type lgssc unregistered [ 9374.688126] LNet: 274990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9374.700813] LNetError: 274990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9374.725771] LNet: Removed LNI 192.168.202.108@tcp [ 9375.574403] Key type .llcrypt unregistered [ 9375.576851] Key type ._llcrypt unregistered [ 9405.103827] Key type ._llcrypt registered [ 9405.105875] Key type .llcrypt registered [ 9405.242438] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 9420.851099] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 9421.987648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9422.174432] alg: No test for adler32 (adler32-zlib) [ 9423.218829] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 9423.463133] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 9425.088162] Key type lgssc registered [ 9426.487182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9481.900041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9481.984237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.264933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9483.303979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9483.399149] Lustre: lustre-MDT0000: new disk, initializing [ 9483.503755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.525154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9489.287537] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9504.368294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.510872] Lustre: 278788:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9504.575609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9504.579587] Lustre: Skipped 1 previous similar message [ 9504.637635] Lustre: lustre-MDT0001: new disk, initializing [ 9504.778565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9504.822856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9504.839988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9510.282703] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9518.960526] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9523.532815] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9532.758339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9533.150869] Lustre: lustre-OST2710: new disk, initializing [ 9533.156580] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9533.255862] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9539.589715] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9540.705407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9540.714675] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9540.882523] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9548.540171] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9556.563895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9556.808332] Lustre: lustre-OST03e8: new disk, initializing [ 9556.811092] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9556.928427] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9564.170320] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9564.251484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9564.257114] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9564.395272] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9570.205647] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9578.986973] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9579.199980] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9584.495190] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9584.833662] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9590.399442] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9590.612207] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9594.890628] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9595.236831] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9602.018466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9602.039274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9602.055039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9605.611187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9605.611409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.712921] Lustre: server umount lustre-MDT0000 complete [ 9612.652709] LustreError: 278780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774884006 with bad export cookie 8587497160077341549 [ 9612.659113] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9612.666288] LustreError: 278780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9613.496336] Lustre: server umount lustre-MDT0001 complete [ 9628.774450] Lustre: server umount lustre-OST2710 complete [ 9631.520099] Lustre: 276584:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774884009/real 1774884009] req@000000007ebf903e x1861100573366400/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774884025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9631.565569] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9631.583261] Lustre: Skipped 2 previous similar messages [ 9632.999635] Lustre: server umount lustre-OST03e8 complete [ 9653.738504] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [ 9664.734400] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 9705.669357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9705.955601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9705.976559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9706.075619] Lustre: lustre-MDT0000: new disk, initializing [ 9706.184497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9706.198853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9711.094702] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9722.009476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.140417] Lustre: 286517:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9722.176902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9722.180308] Lustre: Skipped 1 previous similar message [ 9722.266352] Lustre: lustre-MDT0001: new disk, initializing [ 9722.377665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9722.415919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9722.434444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9726.607744] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9736.092457] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9740.919894] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9750.431897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9750.670928] Lustre: lustre-OST0000: new disk, initializing [ 9750.675718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9750.765345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9757.160724] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9759.809385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9759.820329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9759.949780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9765.206371] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9770.952988] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9771.239448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9776.413425] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9776.638100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9780.707507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9780.711309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9780.719768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9780.727684] Lustre: Skipped 1 previous similar message [ 9780.734791] Lustre: Skipped 4 previous similar messages [ 9784.669744] Lustre: server umount lustre-OST0000 complete [ 9788.901554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9788.907721] LustreError: Skipped 1 previous similar message [ 9788.911267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.929734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9795.098629] Lustre: server umount lustre-MDT0000 complete [ 9796.065877] LustreError: 287853:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.085577] LustreError: 287853:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 6 previous similar messages [ 9799.204673] LustreError: 288270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774884192 with bad export cookie 8587497160077343243 [ 9799.205107] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9799.209299] LustreError: 288270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9799.488564] Lustre: server umount lustre-MDT0001 complete [ 9808.496600] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:23:19 (1774884199) [ 9810.113907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9811.921755] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:23:23 (1774884203) [ 9816.261882] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 9845.912740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9846.113310] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9846.129201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9846.253573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9850.722727] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9858.961850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.082125] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9859.130896] Lustre: 291952:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9864.118371] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [ 9869.906202] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9873.904704] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9894.317980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9894.569738] Lustre: Denying initial registration attempt from nid 192.168.202.108@tcp, specified as failover [ 9894.585215] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9894.593477] LustreError: 293565:0:(tgt_mount.c:2565:server_fill_super()) Unable to start targets: -99 [ 9894.598590] LustreError: 293565:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 9894.611071] LustreError: 293565:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 9894.729493] Lustre: server umount lustre-OST0000 complete [ 9894.732048] LustreError: 293565:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9903.586610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9903.611094] Lustre: Skipped 1 previous similar message [ 9903.627772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9903.630861] Lustre: Skipped 2 previous similar messages [ 9909.310435] Lustre: server umount lustre-MDT0000 complete [ 9913.792964] LustreError: 291945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774884307 with bad export cookie 8587497160077344125 [ 9913.805323] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.818985] LustreError: 291945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9914.340559] Lustre: server umount lustre-MDT0001 complete [ 9926.535348] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [ 9929.792239] Key type lgssc unregistered [ 9930.243737] LNet: 294670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9930.252625] LNetError: 294670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9930.280313] LNet: Removed LNI 192.168.202.108@tcp [ 9931.158266] Key type .llcrypt unregistered [ 9931.169373] Key type ._llcrypt unregistered [ 9943.359611] Key type ._llcrypt registered [ 9943.363721] Key type .llcrypt registered [ 9943.462599] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [ 9944.479821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9944.576960] alg: No test for adler32 (adler32-zlib) [ 9945.813494] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 9946.120597] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [ 9947.872524] Key type lgssc registered [ 9949.225428] Lustre: Echo OBD driver; http://www.lustre.org/ [10011.728930] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [10022.335661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10022.362353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.700551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10023.722527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10023.926599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10028.916239] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10039.015739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.295461] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10039.354556] Lustre: 297410:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10039.696076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10044.286047] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10050.870225] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10054.480872] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10074.749944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10075.082661] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10075.296594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10081.790398] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10088.830340] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10091.490398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10091.491439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10091.498901] LustreError: Skipped 1 previous similar message [10091.506621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10091.527777] Lustre: Skipped 1 previous similar message [10096.619666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10096.629516] Lustre: Skipped 2 previous similar messages [10097.648550] Lustre: server umount lustre-OST0000 complete [10106.337812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10106.341708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.350991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10108.557597] Lustre: server umount lustre-MDT0000 complete [10112.572293] LustreError: 297401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774884506 with bad export cookie 10939353834301814 [10112.574416] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10112.582834] LustreError: 297401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10112.892179] Lustre: server umount lustre-MDT0001 complete [10128.038252] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [10131.233066] Key type lgssc unregistered [10131.604673] LNet: 300631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10131.615732] LNetError: 300631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10131.635911] LNet: Removed LNI 192.168.202.108@tcp [10132.601575] Key type .llcrypt unregistered [10132.603572] Key type ._llcrypt unregistered [10159.969372] Key type ._llcrypt registered [10159.971409] Key type .llcrypt registered [10160.072492] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [10161.641576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10161.684253] alg: No test for adler32 (adler32-zlib) [10162.879326] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [10163.196466] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [10164.865528] Key type lgssc registered [10166.185448] Lustre: Echo OBD driver; http://www.lustre.org/ [10174.932784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10175.014393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10176.560864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10181.736562] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10186.723470] LustreError: 302030:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10186.737751] LustreError: 302030:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10191.844528] LustreError: 302031:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.430939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10192.854060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10197.095059] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10204.120077] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10208.089680] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10217.982905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10218.524773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10224.699847] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10231.432451] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10248.171383] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.176358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10248.195131] Lustre: Skipped 2 previous similar messages [10248.196723] Lustre: Skipped 1 previous similar message [10250.279356] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.8@tcp (stopping) [10250.285762] Lustre: Skipped 1 previous similar message [10253.241675] Lustre: server umount lustre-MDT0000 complete [10253.301859] LustreError: 303281:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.329338] LustreError: 303281:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10255.371653] LustreError: 304081:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.784876] LustreError: 302010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774884651 with bad export cookie 8455671584648392677 [10257.789742] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10257.796226] LustreError: 302010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10258.403357] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.415664] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10260.480481] LustreError: 304189:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10260.505446] LustreError: 304189:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10260.524123] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [10265.606096] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [10265.617059] Lustre: Skipped 1 previous similar message [10270.744340] LustreError: 302024:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.763019] LustreError: 302024:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [10272.224099] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10272.468697] Lustre: server umount lustre-MDT0001 complete [10280.438988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10290.837030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.498589] LustreError: 301340:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e8739b63 x1861101348235776/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10302.992596] LustreError: 303638:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10303.013767] LustreError: 303638:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10303.117040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10303.200489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10309.188376] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10320.285898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10321.101762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10323.185205] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10325.170508] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10333.491980] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10336.056615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10341.207739] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10344.486264] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10354.155444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.165593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10354.174952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10354.184685] Lustre: Skipped 4 previous similar messages [10359.272875] LustreError: 303629:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.302929] LustreError: 303629:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [10359.463894] Lustre: server umount lustre-OST0000 complete [10364.385382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10364.404164] Lustre: Skipped 2 previous similar messages [10370.499351] Lustre: server umount lustre-MDT0000 complete [10374.728583] LustreError: 304503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774884768 with bad export cookie 8455671584648400335 [10374.739955] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10374.748510] LustreError: 304503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10375.100856] Lustre: server umount lustre-MDT0001 complete [10387.562533] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [10391.917343] Key type lgssc unregistered [10392.478793] LNet: 307751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10392.493239] LNetError: 307751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10392.524505] LNet: Removed LNI 192.168.202.108@tcp [10393.913450] Key type .llcrypt unregistered [10393.918310] Key type ._llcrypt unregistered [10425.563247] Key type ._llcrypt registered [10425.566190] Key type .llcrypt registered [10425.725668] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [10426.775242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10426.812095] alg: No test for adler32 (adler32-zlib) [10428.044068] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [10428.272728] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [10429.912212] Key type lgssc registered [10431.385816] Lustre: Echo OBD driver; http://www.lustre.org/ [10440.714386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10440.742915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.374336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10447.354842] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10452.973439] LustreError: 309151:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.001598] LustreError: 309151:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [10456.316517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10456.710442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10461.475510] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10468.307944] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10473.340115] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10477.028352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10477.050945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10477.093620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10481.635197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10481.654581] Lustre: Skipped 2 previous similar messages [10486.753811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10486.760309] Lustre: Skipped 1 previous similar message [10491.360868] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10491.631126] Lustre: server umount lustre-MDT0000 complete [10491.874503] LustreError: 309862:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.901562] LustreError: 309862:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10495.728364] LustreError: 309132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774884889 with bad export cookie 12299628399963433453 [10495.739755] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10495.747035] LustreError: 309132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10496.134398] Lustre: server umount lustre-MDT0001 complete [10511.915032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.236305] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10512.268496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10512.501199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10517.192460] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10527.462265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.634785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10527.697434] Lustre: 311516:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10528.044058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10533.439875] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10540.337109] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10545.898790] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10559.289137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10559.560482] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10559.568290] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10559.573483] LustreError: 313173:0:(tgt_mount.c:2565:server_fill_super()) Unable to start targets: -2 [10559.577610] LustreError: 313173:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [10559.581633] LustreError: 313173:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [10559.674516] Lustre: server umount lustre-OST0000 complete [10559.676504] LustreError: 313173:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [10572.024555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10572.395872] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10572.664361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10578.135537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10580.267067] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10586.374936] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10594.982784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10595.133364] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10595.240281] Lustre: lustre-OST0001: new disk, initializing [10595.243259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10595.328083] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10602.780626] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10603.552083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10603.561516] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10603.628216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10610.077776] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10613.733973] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10613.742924] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10613.757699] Lustre: Skipped 1 previous similar message [10613.765412] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10613.769237] Lustre: Skipped 1 previous similar message [10618.855907] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10618.866261] Lustre: Skipped 1 previous similar message [10619.204952] Lustre: server umount lustre-OST0001 complete [10623.973806] LustreError: 313639:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10623.994446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.001850] LustreError: 313639:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10624.022338] Lustre: Skipped 1 previous similar message [10629.145587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10629.146457] LustreError: 313639:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.158778] Lustre: Skipped 3 previous similar messages [10629.191767] LustreError: 313639:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10634.214260] LustreError: 313640:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10634.253855] LustreError: 313640:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10638.304125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10638.562239] Lustre: server umount lustre-OST0000 complete [10644.450286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10644.486773] Lustre: Skipped 2 previous similar messages [10649.576018] LustreError: 311523:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10649.601085] LustreError: 311523:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [10649.767829] Lustre: server umount lustre-MDT0000 complete [10654.480136] LustreError: 311509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774885048 with bad export cookie 12299628399963433999 [10654.483481] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.493604] LustreError: 311509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10655.124503] Lustre: server umount lustre-MDT0001 complete [10668.400740] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [10672.834610] Key type lgssc unregistered [10673.280553] LNet: 316434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10673.294613] LNetError: 316434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10673.317344] LNet: Removed LNI 192.168.202.108@tcp [10674.557567] Key type .llcrypt unregistered [10674.562571] Key type ._llcrypt unregistered [10762.324305] Key type ._llcrypt registered [10762.330388] Key type .llcrypt registered [10762.473632] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [10779.650790] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [10781.111970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10781.262099] alg: No test for adler32 (adler32-zlib) [10782.372469] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [10782.641833] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [10784.376703] Key type lgssc registered [10785.570758] Lustre: Echo OBD driver; http://www.lustre.org/ [10827.895289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10827.950644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.222488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10829.252930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10829.373891] Lustre: lustre-MDT0000: new disk, initializing [10829.498374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10829.531578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10835.060623] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10846.907820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10847.043121] Lustre: 321342:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10847.107713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10847.115198] Lustre: Skipped 1 previous similar message [10847.221434] Lustre: lustre-MDT0001: new disk, initializing [10847.337290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10847.389707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10847.409145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10851.869226] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10861.256670] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10866.076234] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10874.222159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10874.658927] Lustre: lustre-OST0000: new disk, initializing [10874.664612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10874.813139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10882.346227] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10882.687116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10882.702696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10882.810849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10891.500209] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10897.745662] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10898.058254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10903.539139] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10903.871602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10908.657267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10908.658524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10908.683107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10908.689381] Lustre: Skipped 1 previous similar message [10908.733203] LustreError: Skipped 1 previous similar message [10912.508160] Lustre: server umount lustre-OST0000 complete [10918.883333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.884101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10918.907809] Lustre: Skipped 1 previous similar message [10918.910656] Lustre: Skipped 1 previous similar message [10918.931922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10923.433304] Lustre: server umount lustre-MDT0000 complete [10927.761948] LustreError: 321333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774885321 with bad export cookie 4227750142240794097 [10927.771825] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10928.413281] Lustre: server umount lustre-MDT0001 complete [10939.143596] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:42:10 (1774885330) [10946.800502] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:42:18 (1774885338) [10955.215641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.766769] LustreError: 325692:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.792584] LustreError: 325692:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [10955.891100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10961.298746] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10961.387750] LustreError: 325693:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10965.479347] LustreError: 325692:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.595841] LustreError: 325693:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.128290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10971.522399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10976.222961] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [10982.797405] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10987.810771] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10996.992353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10997.161583] LustreError: 327287:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10997.324937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11005.095876] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11012.867608] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11021.145195] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11023.290267] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11025.442218] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11028.454155] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11031.032711] Lustre: Failing over lustre-MDT0000 [11031.371333] Lustre: server umount lustre-MDT0000 complete [11031.524810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11031.535153] LustreError: 325689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.555312] Lustre: Skipped 2 previous similar messages [11031.588472] LustreError: 325689:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [11050.510046] LustreError: 325689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11050.550555] LustreError: 325689:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 12 previous similar messages [11052.114863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11052.305560] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11052.609931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11052.667280] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11055.628576] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11057.564816] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11057.647745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11057.697019] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11057.759128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11066.959552] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11068.889645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11073.102073] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11075.622114] Lustre: Failing over lustre-MDT0000 [11075.934947] Lustre: server umount lustre-MDT0000 complete [11076.080415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.112302] Lustre: Skipped 1 previous similar message [11086.307318] LustreError: 325689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.326833] LustreError: 325689:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 13 previous similar messages [11091.424397] Lustre: 319563:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774885469/real 1774885469] req@000000006fc98748 x1861101998400768/t0(0) o400->MGC192.168.202.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1774885485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11091.459544] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11096.505210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.665858] LustreError: 319561:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004a744772 x1861101998405632/t0(0) o250->MGC192.168.202.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11102.177981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11102.261681] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11106.819640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11107.337126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11107.347325] Lustre: Skipped 2 previous similar messages [11107.374405] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11107.431266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11108.379431] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11121.822890] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11124.473810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11132.908496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11132.926177] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11132.935551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11132.935559] Lustre: Skipped 1 previous similar message [11132.991447] Lustre: Skipped 3 previous similar messages [11138.023380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11138.036798] Lustre: Skipped 4 previous similar messages [11138.709382] Lustre: server umount lustre-MDT0000 complete [11143.514902] LustreError: 327309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774885537 with bad export cookie 4227750142240796806 [11143.516814] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11143.522506] LustreError: 327309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11144.055718] Lustre: server umount lustre-MDT0001 complete [11159.975446] Lustre: server umount lustre-OST0000 complete [11180.148620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11180.522449] LustreError: 331485:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11180.551661] LustreError: 331485:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 17 previous similar messages [11180.619359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11186.498766] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11196.089570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11202.371697] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11210.673742] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11216.450925] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11224.560133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11225.978805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11232.621408] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11240.428658] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11256.804273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11256.812396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11256.813277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11256.829522] Lustre: Skipped 1 previous similar message [11266.016717] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11266.315949] Lustre: server umount lustre-OST0000 complete [11272.164848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11272.171632] Lustre: Skipped 4 previous similar messages [11277.330993] Lustre: server umount lustre-MDT0000 complete [11282.793963] LustreError: 331467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774885676 with bad export cookie 4227750142240797555 [11282.808168] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11283.762741] Lustre: server umount lustre-MDT0001 complete [11299.150836] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [11302.612121] Key type lgssc unregistered [11302.976579] LNet: 334669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11302.997045] LNetError: 334669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11303.019327] LNet: Removed LNI 192.168.202.108@tcp [11303.938791] Key type .llcrypt unregistered [11303.940462] Key type ._llcrypt unregistered [11337.079331] Key type ._llcrypt registered [11337.083078] Key type .llcrypt registered [11337.196941] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [11338.388121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11338.510118] alg: No test for adler32 (adler32-zlib) [11339.765926] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [11340.034836] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [11341.738444] Key type lgssc registered [11343.491682] Lustre: Echo OBD driver; http://www.lustre.org/ [11354.475666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11354.495754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.085260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11361.544734] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11365.348443] LustreError: 336168:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11365.379611] LustreError: 336168:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [11370.472960] LustreError: 336169:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11372.002618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11372.486766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11378.521559] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11386.261189] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11391.519770] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11400.993669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11401.190876] LustreError: 337765:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.620773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11406.838930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11409.240375] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11415.817979] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11428.323200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11428.341661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11428.352126] Lustre: Skipped 2 previous similar messages [11428.370983] Lustre: Skipped 1 previous similar message [11430.234531] Lustre: server umount lustre-MDT0000 complete [11433.455376] LustreError: 336163:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11433.489089] LustreError: 336163:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [11435.211737] LustreError: 336149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774885828 with bad export cookie 16736023568530235811 [11435.214302] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11435.229172] LustreError: 336149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11435.708845] Lustre: server umount lustre-MDT0001 complete [11451.272392] Lustre: server umount lustre-OST0000 complete [11470.651588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.481776] LustreError: 339548:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.578183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11477.316042] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11482.094337] LustreError: 339549:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.128043] LustreError: 339549:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [11487.997612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11488.308111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11493.006630] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11501.499479] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11506.574470] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11516.884961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11517.452915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11526.078499] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11534.307883] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11542.033231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11562.467482] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11562.474330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11562.499418] Lustre: Skipped 2 previous similar messages [11562.519307] Lustre: Skipped 3 previous similar messages [11567.275795] Lustre: server umount lustre-MDT0000 complete [11567.586600] LustreError: 339549:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.635471] LustreError: 339549:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [11572.386265] LustreError: 339529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774885966 with bad export cookie 16736023568530236924 [11572.410962] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11572.712427] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11572.838181] Lustre: server umount lustre-MDT0001 complete [11578.038981] Lustre: server umount lustre-OST0000 complete [11600.267640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11600.956160] LustreError: 343145:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.982868] LustreError: 343145:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [11601.127886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11606.945440] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11618.083742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11624.521193] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11631.816184] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11637.147680] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11646.007338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11646.508436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11646.518891] Lustre: Skipped 1 previous similar message [11655.012093] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11663.430513] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11671.105651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11681.252862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11681.254819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11681.289970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.311568] Lustre: Skipped 1 previous similar message [11685.567539] Lustre: server umount lustre-OST0000 complete [11690.480082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11690.497975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11690.522123] Lustre: Skipped 1 previous similar message [11690.532549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11696.609898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11696.625343] Lustre: Skipped 3 previous similar messages [11696.939729] Lustre: server umount lustre-MDT0000 complete [11701.733188] LustreError: 343146:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11701.761493] LustreError: 343146:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [11702.018837] LustreError: 343126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774886095 with bad export cookie 16736023568530237946 [11702.020745] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11702.025315] LustreError: 343126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11702.531418] Lustre: server umount lustre-MDT0001 complete [11716.673254] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [11720.079327] Key type lgssc unregistered [11720.467207] LNet: 346350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11720.480860] LNetError: 346350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11720.507713] LNet: Removed LNI 192.168.202.108@tcp [11721.673127] Key type .llcrypt unregistered [11721.676888] Key type ._llcrypt unregistered [11755.819706] Key type ._llcrypt registered [11755.826487] Key type .llcrypt registered [11756.070521] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [11757.925401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11758.059788] alg: No test for adler32 (adler32-zlib) [11759.457966] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [11759.796862] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [11761.584628] Key type lgssc registered [11763.082643] Lustre: Echo OBD driver; http://www.lustre.org/ [11773.363363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11773.393355] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11773.400750] LustreError: 347777:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11773.415490] LustreError: 347777:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [11773.421196] LustreError: 347777:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11773.430480] LustreError: 347777:0:(tgt_mount.c:2533:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11773.443681] LustreError: 347777:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11782.573733] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11782.578651] LustreError: 348174:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11782.598158] LustreError: 348174:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [11782.603196] LustreError: 348174:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11782.608449] LustreError: 348174:0:(tgt_mount.c:2533:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11782.613842] LustreError: 348174:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11809.443751] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [11813.597245] Key type lgssc unregistered [11813.961717] LNet: 349161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11813.966194] LNetError: 349161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11813.987944] LNet: Removed LNI 192.168.202.108@tcp [11815.040750] Key type .llcrypt unregistered [11815.043560] Key type ._llcrypt unregistered [11839.302102] Key type ._llcrypt registered [11839.304384] Key type .llcrypt registered [11839.480430] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [11856.476970] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [11858.104959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11858.168942] alg: No test for adler32 (adler32-zlib) [11859.274857] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [11859.557938] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [11861.283167] Key type lgssc registered [11862.980738] Lustre: Echo OBD driver; http://www.lustre.org/ [11909.850909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11909.916851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11911.498232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11911.562509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11911.719864] Lustre: lustre-MDT0000: new disk, initializing [11911.856539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11911.886818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11918.513842] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11932.510989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11932.686772] Lustre: 352288:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11932.727582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11932.734332] Lustre: Skipped 1 previous similar message [11932.792852] Lustre: lustre-MDT0001: new disk, initializing [11932.878533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11932.937790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11932.949927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11939.363345] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11949.217108] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11954.306071] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11962.547919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11962.871492] Lustre: lustre-OST0000: new disk, initializing [11962.875964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11962.986867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11971.212302] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [11971.718378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11971.738135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11971.897401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11981.857363] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11988.085689] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11988.443176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11994.483657] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11994.875700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11997.668347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11997.674189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11997.677470] LustreError: Skipped 1 previous similar message [11997.680814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11997.685526] Lustre: Skipped 1 previous similar message [12002.789581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12002.801698] Lustre: Skipped 2 previous similar messages [12004.029585] Lustre: server umount lustre-OST0000 complete [12009.955119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12009.962168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12009.973162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12013.041682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12013.050148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12015.438472] Lustre: server umount lustre-MDT0000 complete [12020.587414] LustreError: 352281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774886414 with bad export cookie 15582797928003258196 [12020.604172] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12020.616293] LustreError: 352281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12021.107736] Lustre: server umount lustre-MDT0001 complete [12031.622735] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12033.529646] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:00:24 (1774886424) [12040.722307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12041.291703] LustreError: 356344:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12041.305684] LustreError: 356344:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [12041.371756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12046.541878] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12046.818450] LustreError: 356345:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12051.942199] LustreError: 356344:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12057.065728] LustreError: 356345:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12058.046268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12058.589643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12064.565949] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12072.544633] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12077.042640] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12084.823794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12085.267780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12093.729079] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12100.948517] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12109.701912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12109.882519] Lustre: lustre-OST0001: new disk, initializing [12109.886474] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12110.001077] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12116.476658] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12118.596781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12118.608462] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12118.678592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12126.397756] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12133.925933] Lustre: lustre-OST0001: Not available for connect from 192.168.202.8@tcp (stopping) [12133.933055] Lustre: Skipped 1 previous similar message [12134.377289] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12134.394724] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12137.451669] Lustre: server umount lustre-OST0001 complete [12139.028448] LustreError: 357952:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.743833] LustreError: 357953:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.768810] LustreError: 357953:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [12157.922134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12157.930207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.953689] Lustre: Skipped 1 previous similar message [12157.964122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12157.977956] Lustre: Skipped 2 previous similar messages [12162.292297] Lustre: server umount lustre-OST0000 complete [12170.215314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12170.230218] Lustre: Skipped 2 previous similar messages [12173.858687] Lustre: server umount lustre-MDT0000 complete [12175.339924] LustreError: 356340:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12175.360823] LustreError: 356340:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [12178.749896] LustreError: 356326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774886572 with bad export cookie 15582797928003259085 [12178.762028] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12178.764920] LustreError: 356326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12193.254959] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12193.682325] Lustre: server umount lustre-MDT0001 complete [12206.639912] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [12210.932385] Key type lgssc unregistered [12211.406928] LNet: 360718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12211.428401] LNetError: 360718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12211.451847] LNet: Removed LNI 192.168.202.108@tcp [12212.690539] Key type .llcrypt unregistered [12212.696091] Key type ._llcrypt unregistered [12277.367404] Key type ._llcrypt registered [12277.374135] Key type .llcrypt registered [12277.590601] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [12279.296838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12279.509231] alg: No test for adler32 (adler32-zlib) [12280.783500] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [12281.180986] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [12282.993265] Key type lgssc registered [12284.717729] Lustre: Echo OBD driver; http://www.lustre.org/ [12295.271931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12295.301801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12296.837727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12296.956847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12297.277224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12302.154517] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12314.150489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.461224] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12314.557509] Lustre: 363504:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12315.147757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12322.214212] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12329.891551] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12334.579831] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12337.142652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12337.160073] Lustre: Skipped 1 previous similar message [12337.169362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12337.171768] Lustre: Skipped 1 previous similar message [12342.253237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12342.264751] Lustre: Skipped 1 previous similar message [12343.627868] Lustre: server umount lustre-MDT0000 complete [12348.622921] LustreError: 363494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774886742 with bad export cookie 10006307103218782555 [12348.626877] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12348.631990] LustreError: 363494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12348.904300] Lustre: server umount lustre-MDT0001 complete [12361.131914] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:05:51 (1774886751) [12370.838266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12371.720487] LustreError: 365790:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12371.743831] LustreError: 365790:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [12371.810691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12377.070178] LustreError: 365791:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12377.665922] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12382.182865] LustreError: 365790:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12387.303912] LustreError: 365791:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12389.072508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12389.718988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12395.926297] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12403.366952] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12407.709660] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12417.034806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12417.313105] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12417.601476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12426.027778] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12433.053806] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12442.873568] Lustre: Permanently deactivating lustre-OST0000 [12442.897866] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12451.493572] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12453.489452] LustreError: 368439:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12453.506913] LustreError: 368439:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12453.519415] LustreError: 368439:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12456.878274] LustreError: 368487:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12456.883831] LustreError: 368487:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12465.332108] Lustre: server umount lustre-OST0000 complete [12469.619232] LustreError: 368737:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12469.623868] LustreError: 368737:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12475.362119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12475.380241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12480.488143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12480.503805] Lustre: Skipped 2 previous similar messages [12481.596362] Lustre: server umount lustre-MDT0000 complete [12485.601833] LustreError: 367307:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12485.634882] LustreError: 367307:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [12486.108929] LustreError: 366505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774886879 with bad export cookie 10006307103218783136 [12486.116236] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12486.124964] LustreError: 366505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12495.508680] Lustre: server umount lustre-MDT0001 complete [12507.184681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12512.536864] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12516.171218] LustreError: 369881:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12524.710976] Lustre: 370221:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.108@tcp [12527.717674] Lustre: server umount MGS complete [12542.340820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12542.800660] LustreError: 370946:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12542.821171] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12542.827186] Lustre: Skipped 1 previous similar message [12542.908702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12548.388288] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12559.371521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12559.719695] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12559.816455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12565.541177] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12571.612496] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12577.689734] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12580.505758] Lustre: Permanently reactivating lustre-OST0000 [12580.548246] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12580.554689] Lustre: Skipped 2 previous similar messages [12610.530452] LustreError: 372501:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12621.060792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12621.631473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12629.701745] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12636.135678] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12643.844496] Lustre: server umount lustre-OST0000 complete [12651.490937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12651.496468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12651.517897] Lustre: Skipped 2 previous similar messages [12651.537322] Lustre: Skipped 1 previous similar message [12657.719371] Lustre: server umount lustre-MDT0000 complete [12661.710372] Lustre: 362826:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774887039/real 1774887039] req@00000000e3fe7292 x1861103569179136/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1774887055 ref 1 fl Rpc:EXQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [12661.712900] LustreError: 370927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887055 with bad export cookie 10006307103218784186 [12661.717654] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12661.768504] LustreError: 370927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12672.220310] Lustre: server umount lustre-MDT0001 complete [12683.665596] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [12686.868827] Key type lgssc unregistered [12687.192163] LNet: 374959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12687.220790] LNetError: 374959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12688.298083] LNet: Removed LNI 192.168.202.108@tcp [12689.496831] Key type .llcrypt unregistered [12689.501590] Key type ._llcrypt unregistered [12715.339500] Key type ._llcrypt registered [12715.340946] Key type .llcrypt registered [12715.492288] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [12732.066640] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [12733.616581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12733.773025] alg: No test for adler32 (adler32-zlib) [12735.058905] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [12735.442863] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [12737.298425] Key type lgssc registered [12739.029950] Lustre: Echo OBD driver; http://www.lustre.org/ [12782.231464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12782.303108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12783.807939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12783.882791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12784.011606] Lustre: lustre-MDT0000: new disk, initializing [12784.199751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12784.243917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12790.791488] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12806.531966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12806.735595] Lustre: 378084:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12806.813067] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12806.825187] Lustre: Skipped 1 previous similar message [12806.956789] Lustre: lustre-MDT0001: new disk, initializing [12807.219587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12807.276108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12807.292209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12813.193982] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12822.997210] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12828.711454] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12837.065029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12837.418721] Lustre: lustre-OST0000: new disk, initializing [12837.422192] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12837.540683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12843.077689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12843.087548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12843.241044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12845.460956] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12854.537738] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12860.281633] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12860.562222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12865.776398] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12866.011153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12869.093124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12869.111787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12869.142772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12874.212754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12874.232102] Lustre: Skipped 1 previous similar message [12874.696259] Lustre: server umount lustre-OST0000 complete [12878.819101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12878.823822] LustreError: Skipped 1 previous similar message [12878.853169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12878.864589] Lustre: Skipped 1 previous similar message [12878.871652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12878.881318] Lustre: Skipped 1 previous similar message [12884.451830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12884.464846] Lustre: Skipped 3 previous similar messages [12884.808827] Lustre: server umount lustre-MDT0000 complete [12889.003285] LustreError: 379827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887282 with bad export cookie 12116912915501942669 [12889.007126] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12889.014357] LustreError: 379827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12889.629145] Lustre: server umount lustre-MDT0001 complete [12899.764194] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12901.643312] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:14:52 (1774887292) [12910.088400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12910.757145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12916.204930] LustreError: 382143:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12916.224997] LustreError: 382143:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [12916.538463] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12920.294589] LustreError: 382144:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12925.422854] LustreError: 382143:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12927.274985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12928.018064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12932.912196] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12939.740836] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12944.444650] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12952.556470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12952.838495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12960.094920] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [12967.161923] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12969.510529] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12972.677315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12986.849920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12986.862554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.881800] Lustre: Skipped 1 previous similar message [12986.886639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12989.402835] Lustre: server umount lustre-OST0000 complete [12994.530895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12994.535929] LustreError: Skipped 1 previous similar message [12994.539286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12994.552972] Lustre: Skipped 1 previous similar message [12998.136881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12998.159637] Lustre: Skipped 4 previous similar messages [13008.352627] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13008.637618] Lustre: server umount lustre-MDT0000 complete [13012.451053] LustreError: 382144:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13012.478147] LustreError: 382144:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13012.966146] LustreError: 382124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887406 with bad export cookie 12116912915501943551 [13012.971157] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13012.977054] LustreError: 382124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13013.393572] Lustre: server umount lustre-MDT0001 complete [13025.877458] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [13028.989066] Key type lgssc unregistered [13029.370141] LNet: 385520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13029.380163] LNetError: 385520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13029.400388] LNet: Removed LNI 192.168.202.108@tcp [13030.546046] Key type .llcrypt unregistered [13030.550400] Key type ._llcrypt unregistered [13060.415797] Key type ._llcrypt registered [13060.421074] Key type .llcrypt registered [13060.536953] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [13060.837834] Key type .llcrypt unregistered [13060.843089] Key type ._llcrypt unregistered [13080.151975] Key type ._llcrypt registered [13080.154304] Key type .llcrypt registered [13080.271728] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [13081.241687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13081.335499] alg: No test for adler32 (adler32-zlib) [13082.900510] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [13083.138824] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [13084.768255] Key type lgssc registered [13087.145280] Lustre: Echo OBD driver; http://www.lustre.org/ [13100.230787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13100.292199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13102.242790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13108.218991] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13111.780931] LustreError: 387701:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13111.804019] LustreError: 387701:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [13116.903288] LustreError: 387702:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13121.447906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13121.952764] LustreError: 387701:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13122.004074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13127.142013] LustreError: 387701:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13127.177016] LustreError: 387701:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [13127.202488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13129.406194] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13132.267640] LustreError: 387702:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13132.320241] LustreError: 387702:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13136.368544] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13142.502612] LustreError: 387701:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13142.543484] LustreError: 387701:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [13145.165159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13145.619382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13151.593970] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13162.979430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13162.994377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13163.020715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13166.053228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13166.062545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13166.063157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13167.913647] Lustre: server umount lustre-OST0000 complete [13175.885970] Lustre: server umount lustre-MDT0000 complete [13176.291978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13176.297681] LustreError: 389504:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13176.310435] Lustre: Skipped 1 previous similar message [13176.331219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13179.974995] LustreError: 387681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887573 with bad export cookie 11881589397904264521 [13179.991135] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13180.028980] LustreError: 387011:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting [13180.708826] Lustre: server umount lustre-MDT0001 complete [13194.373542] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [13197.914279] Key type lgssc unregistered [13198.206637] LNet: 390689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13198.214284] LNetError: 390689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13198.233248] LNet: Removed LNI 192.168.202.108@tcp [13199.111253] Key type .llcrypt unregistered [13199.114243] Key type ._llcrypt unregistered [13227.002243] Key type ._llcrypt registered [13227.008505] Key type .llcrypt registered [13227.128640] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [13228.957104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13228.991256] alg: No test for adler32 (adler32-zlib) [13230.058944] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [13230.298963] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [13232.049105] Key type lgssc registered [13233.384601] Lustre: Echo OBD driver; http://www.lustre.org/ [13243.098222] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13243.123142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13259.040623] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13264.160602] LustreError: 392075:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.108@tcp: failed processing log, type 4: rc = -110 [13290.912236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13298.472460] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13307.746602] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13318.716302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13319.527557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13319.641081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13324.784972] LustreError: 392101:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13324.807684] LustreError: 392101:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [13325.460378] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13326.830158] LustreError: 392100:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13326.861865] LustreError: 392100:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13331.954845] LustreError: 393001:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13331.977722] LustreError: 393001:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13337.066319] LustreError: 393001:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13337.105595] LustreError: 393001:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13337.208440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13337.773402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13343.253197] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13350.376377] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13356.743440] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13365.729756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13365.744914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13365.769161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13368.293193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13368.305196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13368.338554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13371.516360] Lustre: server umount lustre-OST0000 complete [13376.238682] Lustre: server umount lustre-MDT0000 complete [13378.529724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13378.554180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13378.573286] LustreError: 393005:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13380.824502] LustreError: 392083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887774 with bad export cookie 6419690310599152046 [13380.853545] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13381.321813] Lustre: server umount lustre-MDT0001 complete [13394.253918] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [13397.438555] Key type lgssc unregistered [13397.800480] LNet: 395292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13397.808910] LNetError: 395292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13397.834650] LNet: Removed LNI 192.168.202.108@tcp [13398.901346] Key type .llcrypt unregistered [13398.902803] Key type ._llcrypt unregistered [13433.202671] Key type ._llcrypt registered [13433.209689] Key type .llcrypt registered [13433.439459] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [13435.416272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13435.495033] alg: No test for adler32 (adler32-zlib) [13437.811872] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [13438.420867] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [13440.384255] Key type lgssc registered [13442.207122] Lustre: Echo OBD driver; http://www.lustre.org/ [13455.817146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13455.839366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13457.564433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13463.511025] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13468.137752] LustreError: 396693:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13468.173110] LustreError: 396693:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [13473.261050] LustreError: 396694:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13474.242832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13474.897787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13480.178694] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13486.978942] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13491.420740] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13500.746713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13501.373941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13502.427465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13509.031468] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13515.824531] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13520.876701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13520.891559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13520.919370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13522.637254] Lustre: server umount lustre-MDT0000 complete [13522.995160] LustreError: 396690:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13524.964133] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13525.007873] Lustre: Skipped 1 previous similar message [13525.053261] LustreError: 396690:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13525.074632] LustreError: 396690:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [13526.013320] LustreError: 396673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.8@tcp arrived at 1774887919 with bad export cookie 9277464047290194055 [13528.779975] LustreError: 396674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774887922 with bad export cookie 9277464047290193166 [13528.789515] LustreError: 396674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13528.796681] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13530.082319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13530.083964] LustreError: 396693:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13530.097506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13530.101670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13530.138164] LustreError: 396693:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13533.091958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13535.140796] Lustre: server umount lustre-OST0000 complete [13538.276742] LustreError: 396689:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13538.302748] LustreError: 396689:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [13544.312606] Lustre: server umount lustre-MDT0001 complete [13556.282442] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [13559.863845] Key type lgssc unregistered [13560.288265] LNet: 400027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13560.295141] LNetError: 400027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13560.315980] LNet: Removed LNI 192.168.202.108@tcp [13561.452578] Key type .llcrypt unregistered [13561.459222] Key type ._llcrypt unregistered [13593.855506] Key type ._llcrypt registered [13593.859880] Key type .llcrypt registered [13594.052947] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [13595.999976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13596.075787] alg: No test for adler32 (adler32-zlib) [13597.382590] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [13597.763308] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [13599.561505] Key type lgssc registered [13601.561516] Lustre: Echo OBD driver; http://www.lustre.org/ [13611.892921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13611.911146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13613.586408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13619.242779] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13623.783103] LustreError: 401428:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13623.810415] LustreError: 401428:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [13628.703299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13628.914827] LustreError: 401429:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13629.227300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13634.393718] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13642.044577] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13647.303081] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13657.003982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13657.771457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13658.835127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13666.246488] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13674.288503] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13680.616975] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13680.628648] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13680.650044] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13682.225925] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [13684.708305] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13684.727385] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13684.738275] Lustre: Skipped 1 previous similar message [13686.242671] Lustre: server umount lustre-MDT0001 complete [13687.299328] LustreError: 401425:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13689.828087] LustreError: 401424:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13689.847860] LustreError: 401424:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13694.946175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13694.946928] LustreError: 402142:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13694.959451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13694.960511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13699.346717] Lustre: server umount lustre-OST0000 complete [13704.451802] Lustre: server umount lustre-MDT0000 complete [13720.783326] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [13724.579180] Key type lgssc unregistered [13725.057682] LNet: 404766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13725.070828] LNetError: 404766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13725.104677] LNet: Removed LNI 192.168.202.108@tcp [13726.429871] Key type .llcrypt unregistered [13726.431637] Key type ._llcrypt unregistered [13757.554112] Key type ._llcrypt registered [13757.557755] Key type .llcrypt registered [13757.782026] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [13774.127413] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [13775.743568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13775.796363] alg: No test for adler32 (adler32-zlib) [13776.869957] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [13777.085318] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [13778.744240] Key type lgssc registered [13780.316610] Lustre: Echo OBD driver; http://www.lustre.org/ [13828.553420] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [13849.196231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13849.233594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13850.480361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13850.501138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13850.559614] Lustre: lustre-MDT0000: new disk, initializing [13850.626575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13850.679277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13855.816458] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13874.241236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13874.949828] Lustre: lustre-OST0000: new disk, initializing [13874.981281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13874.999803] Lustre: Skipped 1 previous similar message [13875.145708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13883.057970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13883.076345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13883.163760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13885.325473] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13895.640136] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13916.026529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13916.166562] Lustre: 408734:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13916.229041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13916.298979] Lustre: lustre-MDT0001: new disk, initializing [13916.485875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13916.588140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13916.618866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13922.428277] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [13934.015416] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13934.236990] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13938.886409] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13939.045827] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13944.347723] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [13948.388103] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13948.392190] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13948.411891] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13949.443190] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.8@tcp (stopping) [13949.460042] Lustre: Skipped 1 previous similar message [13950.620642] Lustre: server umount lustre-MDT0001 complete [13958.629138] LustreError: 408741:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13958.643556] LustreError: 408741:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 5 previous similar messages [13963.747141] LustreError: 408741:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13963.791122] LustreError: 408741:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13968.884524] LustreError: 408746:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13968.911134] LustreError: 408746:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [13970.402338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13970.410532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13970.420507] Lustre: Skipped 1 previous similar message [13970.433054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13973.990784] LustreError: 408747:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13976.336531] Lustre: server umount lustre-OST0000 complete [13979.106185] LustreError: 408746:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13987.409609] Lustre: server umount lustre-MDT0000 complete [14002.404584] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [14005.553496] Key type lgssc unregistered [14005.831618] LNet: 413308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14005.837750] LNetError: 413308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14006.890620] LNet: Removed LNI 192.168.202.108@tcp [14007.870663] Key type .llcrypt unregistered [14007.872712] Key type ._llcrypt unregistered [14056.216585] Key type ._llcrypt registered [14056.220542] Key type .llcrypt registered [14056.376641] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [14056.665399] Key type .llcrypt unregistered [14056.667043] Key type ._llcrypt unregistered [14076.368769] Key type ._llcrypt registered [14076.372718] Key type .llcrypt registered [14076.495416] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [14078.410536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14078.601527] alg: No test for adler32 (adler32-zlib) [14079.650282] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [14079.919649] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [14081.640212] Key type lgssc registered [14082.892808] Lustre: Echo OBD driver; http://www.lustre.org/ [14113.622916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14113.650262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14114.936353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14114.967350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14115.060991] Lustre: lustre-MDT0000: new disk, initializing [14115.157085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14115.243524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14120.941034] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [14132.839788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14133.050073] Lustre: 416656:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14133.153357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14133.156888] Lustre: Skipped 1 previous similar message [14133.246380] Lustre: lustre-MDT0001: new disk, initializing [14133.322895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14133.357301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14133.366802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14137.977661] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [14147.052731] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14151.401933] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14158.869512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14159.191215] Lustre: lustre-OST0000: new disk, initializing [14159.216583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14159.304992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14165.967706] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [14169.165967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14169.175747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14169.311638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14174.630523] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14184.420163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14184.432417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14184.440325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14190.056302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14190.076933] Lustre: Skipped 4 previous similar messages [14190.467621] Lustre: server umount lustre-MDT0000 complete [14194.505985] LustreError: 416648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888588 with bad export cookie 5186883648434029689 [14194.521666] LustreError: 416648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14194.523391] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14194.824118] Lustre: server umount lustre-MDT0001 complete [14208.885500] Lustre: server umount lustre-OST0000 complete [14221.729110] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:36:52 (1774888612) [14230.104343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14230.558743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14235.281312] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [14235.625611] LustreError: 420313:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14235.664871] LustreError: 420313:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [14240.750261] LustreError: 420314:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14245.196880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14245.584728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14250.057322] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [14257.208423] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14262.203339] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14283.060449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14283.334876] Lustre: Found index 0 for lustre-OST0000, updating log [14283.585038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14290.951275] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [14299.250752] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14300.988180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14316.519164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14316.530922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14316.535150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14316.579634] Lustre: Skipped 3 previous similar messages [14319.407832] Lustre: server umount lustre-OST0000 complete [14338.022982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14338.033951] LustreError: Skipped 1 previous similar message [14338.049170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14338.070750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14338.078897] Lustre: Skipped 3 previous similar messages [14343.161418] Lustre: server umount lustre-MDT0000 complete [14343.656523] LustreError: 420309:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14343.676527] LustreError: 420309:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [14347.970933] LustreError: 420293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774888741 with bad export cookie 5186883648434031082 [14347.977281] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14347.982377] LustreError: 420293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14348.668881] Lustre: server umount lustre-MDT0001 complete [14355.424074] LNet: 1 local NIs in recovery (showing 1): 192.168.202.108@tcp [14360.710720] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:39:11 (1774888751) [14986.500336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14987.002690] LustreError: 426413:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14987.018697] LustreError: 426413:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [14987.079595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14990.692677] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [14992.355165] LustreError: 426414:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14996.449760] LustreError: 426413:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14998.206557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14998.626682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15002.777501] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15008.102406] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15011.840321] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15031.298257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15031.507445] Lustre: Found index 0 for lustre-OST0000, updating log [15031.735595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15034.802705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [15037.548836] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15043.009269] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15045.090425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15045.095285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15045.103301] Lustre: Skipped 1 previous similar message [15045.110665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15045.116463] Lustre: Skipped 2 previous similar messages [15046.181360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15046.688202] LNet: 1 local NIs in recovery (showing 1): 192.168.202.108@tcp [15051.212951] Lustre: server umount lustre-OST0000 complete [15054.818150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15054.825711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15054.842059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15055.906471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15056.932926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15056.943726] Lustre: Skipped 3 previous similar messages [15060.839682] Lustre: server umount lustre-MDT0000 complete [15061.034468] LustreError: 427124:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15061.056629] LustreError: 427124:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [15064.179703] LustreError: 428049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889457 with bad export cookie 5186883648434032167 [15064.180726] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15064.191767] LustreError: 428049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15064.453378] Lustre: server umount lustre-MDT0001 complete [15092.354370] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [15101.249735] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [15133.027682] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:52:04 (1774889524) [15146.784331] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [15149.144970] Key type lgssc unregistered [15149.412664] LNet: 432634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15149.423624] LNetError: 432634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15250.791255] LNet: Removed LNI 192.168.202.108@tcp [15251.397067] Key type .llcrypt unregistered [15251.399586] Key type ._llcrypt unregistered [15264.465961] Key type ._llcrypt registered [15264.469914] Key type .llcrypt registered [15264.561786] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [15265.588735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15265.614721] alg: No test for adler32 (adler32-zlib) [15266.582038] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [15266.748811] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [15268.392745] Key type lgssc registered [15269.142056] Lustre: Echo OBD driver; http://www.lustre.org/ [15272.122771] LNet: Added LNI 192.168.202.108@tcp100 [8/256/0/180] [15272.819636] LNet: Added LNI 192.168.202.108@tcp101 [8/256/0/180] [15273.561775] LNet: Added LNI 192.168.202.108@tcp102 [8/256/0/180] [15274.310948] LNet: Added LNI 192.168.202.108@tcp103 [8/256/0/180] [15275.097463] LNet: Added LNI 192.168.202.108@tcp104 [8/256/0/180] [15275.850865] LNet: Added LNI 192.168.202.108@tcp105 [8/256/0/180] [15276.459760] LNet: Added LNI 192.168.202.108@tcp106 [8/256/0/180] [15277.186603] LNet: Added LNI 192.168.202.108@tcp107 [8/256/0/180] [15278.005051] LNet: Added LNI 192.168.202.108@tcp108 [8/256/0/180] [15278.734539] LNet: Added LNI 192.168.202.108@tcp109 [8/256/0/180] [15279.417978] LNet: Added LNI 192.168.202.108@tcp110 [8/256/0/180] [15280.147593] LNet: Added LNI 192.168.202.108@tcp111 [8/256/0/180] [15280.796547] LNet: Added LNI 192.168.202.108@tcp112 [8/256/0/180] [15281.445341] LNet: Added LNI 192.168.202.108@tcp113 [8/256/0/180] [15282.196323] LNet: Added LNI 192.168.202.108@tcp114 [8/256/0/180] [15282.872340] LNet: Added LNI 192.168.202.108@tcp115 [8/256/0/180] [15283.676878] LNet: Added LNI 192.168.202.108@tcp116 [8/256/0/180] [15284.285918] LNet: Added LNI 192.168.202.108@tcp117 [8/256/0/180] [15285.110835] LNet: Added LNI 192.168.202.108@tcp118 [8/256/0/180] [15285.774880] LNet: Added LNI 192.168.202.108@tcp119 [8/256/0/180] [15286.415744] LNet: Added LNI 192.168.202.108@tcp120 [8/256/0/180] [15287.066994] LNet: Added LNI 192.168.202.108@tcp121 [8/256/0/180] [15287.751905] LNet: Added LNI 192.168.202.108@tcp122 [8/256/0/180] [15288.343571] LNet: Added LNI 192.168.202.108@tcp123 [8/256/0/180] [15288.935705] LNet: Added LNI 192.168.202.108@tcp124 [8/256/0/180] [15289.551683] LNet: Added LNI 192.168.202.108@tcp125 [8/256/0/180] [15290.242551] LNet: Added LNI 192.168.202.108@tcp126 [8/256/0/180] [15290.887224] LNet: Added LNI 192.168.202.108@tcp127 [8/256/0/180] [15291.504798] LNet: Added LNI 192.168.202.108@tcp128 [8/256/0/180] [15292.119039] LNet: Added LNI 192.168.202.108@tcp129 [8/256/0/180] [15292.733149] LNet: Added LNI 192.168.202.108@tcp130 [8/256/0/180] [15293.257346] LNet: Added LNI 192.168.202.108@tcp131 [8/256/0/180] [15293.950971] LNet: Added LNI 192.168.202.108@tcp132 [8/256/0/180] [15294.559521] LNet: Added LNI 192.168.202.108@tcp133 [8/256/0/180] [15295.130174] LNet: Added LNI 192.168.202.108@tcp134 [8/256/0/180] [15295.630967] LNet: Added LNI 192.168.202.108@tcp135 [8/256/0/180] [15300.324987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15300.338284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15301.451179] Lustre: 435624:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.108@tcp: can't add NID 192.168.202.108@tcp130: rc = -75 [15301.530713] LNetError: 433083:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.108 [15301.542228] LNetError: Protocol error connecting to 192.168.202.108@tcp121 at host 192.168.202.108:988: is it running a compatible version of Lustre? [15301.553177] LNetError: Skipped 2 previous similar messages [15301.601068] LNet: 1 local NIs in recovery (showing 1): 192.168.202.108@tcp121 [15301.626242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15301.653467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15301.728406] Lustre: lustre-MDT0000: new disk, initializing [15301.847048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15301.862276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15302.561071] LNetError: 435646:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.108 [15302.569635] LNetError: 435646:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 2 previous similar messages [15302.573893] LNetError: Protocol error connecting to 192.168.202.108@tcp106 at host 192.168.202.108:988: is it running a compatible version of Lustre? [15304.674205] LNetError: 435644:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.108 [15304.682286] LNetError: 435644:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 1 previous similar message [15304.687274] LNetError: Protocol error connecting to 192.168.202.108@tcp121 at host 192.168.202.108:988: is it running a compatible version of Lustre? [15304.692576] LNetError: Skipped 1 previous similar message [15304.780536] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15308.774635] LNetError: 433082:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.108 [15308.781450] LNetError: Protocol error connecting to 192.168.202.108@tcp121 at host 192.168.202.108:988: is it running a compatible version of Lustre? [15312.634566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15312.756608] Lustre: 435641:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15312.805566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15312.809206] Lustre: Skipped 1 previous similar message [15312.872661] Lustre: lustre-MDT0001: new disk, initializing [15312.901684] Lustre: 436425:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.108@tcp: can't add NID 192.168.202.108@tcp130: rc = -75 [15312.911706] Lustre: 436425:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [15312.996991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15313.028152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15313.037410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15315.322338] Lustre: 436586:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.108@tcp: can't add NID 192.168.202.108@tcp130: rc = -75 [15315.333823] Lustre: 436586:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [15316.668621] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15316.961824] LNetError: 435653:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.108 [15316.969528] LNetError: Protocol error connecting to 192.168.202.108@tcp121 at host 192.168.202.108:988: is it running a compatible version of Lustre? [15322.223310] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15324.868495] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15330.095890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15330.331504] Lustre: lustre-OST0000: new disk, initializing [15330.335253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15330.414905] Lustre: 437391:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.108@tcp: can't add NID 192.168.202.108@tcp130: rc = -75 [15330.418924] Lustre: 437391:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [15330.442512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15332.322652] LNetError: 435645:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.108 [15332.331139] LNetError: Protocol error connecting to 192.168.202.108@tcp106 at host 192.168.202.108:988: is it running a compatible version of Lustre? [15332.347747] LNetError: Skipped 1 previous similar message [15335.166324] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15340.055196] Lustre: 437936:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.108@tcp: can't add NID 192.168.202.108@tcp130: rc = -75 [15340.060260] Lustre: 437936:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [15340.106748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15340.111899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15340.205763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15340.558092] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15350.756598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15350.761377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15350.768630] Lustre: Skipped 1 previous similar message [15350.770950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15350.785519] Lustre: Skipped 1 previous similar message [15352.215629] Lustre: server umount lustre-OST0000 complete [15352.807771] LNetError: 435644:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.108 [15352.817342] LNetError: 435644:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 5 previous similar messages [15352.823512] LNetError: Protocol error connecting to 192.168.202.108@tcp118 at host 192.168.202.108:988: is it running a compatible version of Lustre? [15352.836286] LNetError: Skipped 4 previous similar messages [15355.878515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15355.884228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15355.886627] Lustre: Skipped 1 previous similar message [15361.005702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15361.018179] Lustre: Skipped 1 previous similar message [15361.203229] Lustre: server umount lustre-MDT0000 complete [15363.955929] LustreError: 435632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774889757 with bad export cookie 8068378248555774665 [15363.966600] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15363.971432] LustreError: 435632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15364.336874] Lustre: server umount lustre-MDT0001 complete [15372.482316] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [15374.706147] Key type lgssc unregistered [15375.014382] LNet: 439168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15375.019967] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15375.023603] LNet: Removed LNI 192.168.202.108@tcp [15376.032792] LNet: Removed LNI 192.168.202.108@tcp100 [15376.037510] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15376.046572] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15376.052444] LNet: Removed LNI 192.168.202.108@tcp101 [15377.056266] LNet: Removed LNI 192.168.202.108@tcp102 [15377.060344] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15377.072028] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15377.078582] LNet: Removed LNI 192.168.202.108@tcp103 [15378.144161] LNet: Removed LNI 192.168.202.108@tcp104 [15379.169150] LNet: Removed LNI 192.168.202.108@tcp105 [15379.177188] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15379.182968] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15379.189952] LNet: Removed LNI 192.168.202.108@tcp106 [15380.257141] LNet: Removed LNI 192.168.202.108@tcp107 [15381.280480] LNet: Removed LNI 192.168.202.108@tcp108 [15382.304265] LNet: Removed LNI 192.168.202.108@tcp109 [15383.328188] LNet: Removed LNI 192.168.202.108@tcp110 [15383.332314] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15383.342779] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15383.348518] LNet: Removed LNI 192.168.202.108@tcp111 [15384.352295] LNet: Removed LNI 192.168.202.108@tcp112 [15385.376868] LNet: Removed LNI 192.168.202.108@tcp113 [15386.400171] LNet: Removed LNI 192.168.202.108@tcp114 [15387.424333] LNet: Removed LNI 192.168.202.108@tcp115 [15388.448187] LNet: Removed LNI 192.168.202.108@tcp116 [15389.472248] LNet: Removed LNI 192.168.202.108@tcp117 [15390.496228] LNet: Removed LNI 192.168.202.108@tcp118 [15391.520299] LNet: Removed LNI 192.168.202.108@tcp119 [15391.521944] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15391.526573] LNetError: 439168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15391.532559] LNet: Removed LNI 192.168.202.108@tcp120 [15392.544261] LNet: Removed LNI 192.168.202.108@tcp121 [15393.568175] LNet: Removed LNI 192.168.202.108@tcp122 [15394.592272] LNet: Removed LNI 192.168.202.108@tcp123 [15394.594867] LNet: Removed LNI 192.168.202.108@tcp124 [15394.597278] LNet: Removed LNI 192.168.202.108@tcp125 [15394.601779] LNet: Removed LNI 192.168.202.108@tcp126 [15394.608822] LNet: Removed LNI 192.168.202.108@tcp127 [15394.615405] LNet: Removed LNI 192.168.202.108@tcp128 [15394.622435] LNet: Removed LNI 192.168.202.108@tcp129 [15394.630295] LNet: Removed LNI 192.168.202.108@tcp130 [15394.632793] LNet: Removed LNI 192.168.202.108@tcp131 [15394.636347] LNet: Removed LNI 192.168.202.108@tcp132 [15394.637903] LNet: Removed LNI 192.168.202.108@tcp133 [15394.640405] LNet: Removed LNI 192.168.202.108@tcp134 [15394.651497] LNet: Removed LNI 192.168.202.108@tcp135 [15395.151171] Key type .llcrypt unregistered [15395.152797] Key type ._llcrypt unregistered [15411.993298] Key type ._llcrypt registered [15411.994776] Key type .llcrypt registered [15412.093683] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [15422.825570] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [15423.542060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15423.676934] alg: No test for adler32 (adler32-zlib) [15424.691689] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [15424.895788] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [15426.536157] Key type lgssc registered [15427.486860] Lustre: Echo OBD driver; http://www.lustre.org/ [15451.963085] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:57:23 (1774889843) [15465.178834] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [15467.143265] Key type lgssc unregistered [15467.351782] LNet: 443180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15467.356631] LNetError: 443180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15467.371731] LNet: Removed LNI 192.168.202.108@tcp [15467.966872] Key type .llcrypt unregistered [15467.968299] Key type ._llcrypt unregistered [15480.413803] Key type ._llcrypt registered [15480.415985] Key type .llcrypt registered [15480.504861] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [15481.177268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15481.219795] alg: No test for adler32 (adler32-zlib) [15482.185283] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [15482.396441] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [15484.016205] Key type lgssc registered [15484.818700] Lustre: Echo OBD driver; http://www.lustre.org/ [15491.809586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15491.833637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15493.078328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15493.111779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15493.192167] Lustre: lustre-MDT0000: new disk, initializing [15493.304073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15493.324041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15496.639377] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15504.171030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15504.275415] Lustre: 444354:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15504.314222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15504.319789] Lustre: Skipped 1 previous similar message [15504.371386] Lustre: lustre-MDT0001: new disk, initializing [15504.444321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15504.473593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15504.480920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15507.157869] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15512.980749] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15515.683739] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15520.815371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15521.025547] Lustre: lustre-OST0000: new disk, initializing [15521.031235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15521.099363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15525.333258] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15526.940545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15526.945977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15527.002570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15530.826784] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15533.111711] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing lnet_if_list [15536.566745] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15536.588586] LNet: Added LNI 192.168.202.108@tcp10 [8/256/0/180] [15540.452985] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15540.475264] LNet: Added LNI 192.168.202.108@tcp11 [8/256/0/180] [15542.940979] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15542.960942] LNet: Added LNI 192.168.202.108@tcp12 [8/256/0/180] [15545.683547] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15545.702691] LNet: Added LNI 192.168.202.108@tcp13 [8/256/0/180] [15548.852466] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15548.875828] LNet: Added LNI 192.168.202.108@tcp14 [8/256/0/180] [15552.485594] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15552.513273] LNet: Added LNI 192.168.202.108@tcp15 [8/256/0/180] [15555.532884] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15555.559455] LNet: Added LNI 192.168.202.108@tcp16 [8/256/0/180] [15558.552936] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15558.568269] LNet: Added LNI 192.168.202.108@tcp17 [8/256/0/180] [15561.886645] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15561.911649] LNet: Added LNI 192.168.202.108@tcp18 [8/256/0/180] [15565.318713] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15565.330883] LNet: Added LNI 192.168.202.108@tcp19 [8/256/0/180] [15568.662846] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15568.682346] LNet: Added LNI 192.168.202.108@tcp20 [8/256/0/180] [15572.077501] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15572.096819] LNet: Added LNI 192.168.202.108@tcp21 [8/256/0/180] [15575.628490] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15575.652586] LNet: Added LNI 192.168.202.108@tcp22 [8/256/0/180] [15579.543398] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15579.565655] LNet: Added LNI 192.168.202.108@tcp23 [8/256/0/180] [15583.653909] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15583.676880] LNet: Added LNI 192.168.202.108@tcp24 [8/256/0/180] [15587.911358] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15587.929257] LNet: Added LNI 192.168.202.108@tcp25 [8/256/0/180] [15592.045325] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15592.068538] LNet: Added LNI 192.168.202.108@tcp26 [8/256/0/180] [15596.003573] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15596.020375] LNet: Added LNI 192.168.202.108@tcp27 [8/256/0/180] [15599.787789] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15599.812026] LNet: Added LNI 192.168.202.108@tcp28 [8/256/0/180] [15604.262650] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15604.287289] LNet: Added LNI 192.168.202.108@tcp29 [8/256/0/180] [15608.681823] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15608.714438] LNet: Added LNI 192.168.202.108@tcp30 [8/256/0/180] [15613.786529] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15613.816298] LNet: Added LNI 192.168.202.108@tcp31 [8/256/0/180] [15618.262954] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15618.287323] LNet: Added LNI 192.168.202.108@tcp32 [8/256/0/180] [15623.139835] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15623.170243] LNet: Added LNI 192.168.202.108@tcp33 [8/256/0/180] [15628.672982] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15628.701797] LNet: Added LNI 192.168.202.108@tcp34 [8/256/0/180] [15633.581178] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15633.611621] LNet: Added LNI 192.168.202.108@tcp35 [8/256/0/180] [15639.037986] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15639.064805] LNet: Added LNI 192.168.202.108@tcp36 [8/256/0/180] [15644.563714] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15644.582126] LNet: Added LNI 192.168.202.108@tcp37 [8/256/0/180] [15650.308174] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15650.335956] LNet: Added LNI 192.168.202.108@tcp38 [8/256/0/180] [15655.954285] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15655.970946] LNet: Added LNI 192.168.202.108@tcp39 [8/256/0/180] [15662.162525] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15662.182124] LNet: Added LNI 192.168.202.108@tcp40 [8/256/0/180] [15667.491808] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15667.516707] LNet: Added LNI 192.168.202.108@tcp41 [8/256/0/180] [15673.412640] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15673.442319] LNet: Added LNI 192.168.202.108@tcp42 [8/256/0/180] [15679.339769] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15679.372844] LNet: Added LNI 192.168.202.108@tcp43 [8/256/0/180] [15685.359232] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15685.381074] LNet: Added LNI 192.168.202.108@tcp44 [8/256/0/180] [15691.502985] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15691.522374] LNet: Added LNI 192.168.202.108@tcp45 [8/256/0/180] [15697.513625] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15697.543541] LNet: Added LNI 192.168.202.108@tcp46 [8/256/0/180] [15703.413988] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15703.434236] LNet: Added LNI 192.168.202.108@tcp47 [8/256/0/180] [15709.615553] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15709.669826] LNet: Added LNI 192.168.202.108@tcp48 [8/256/0/180] [15715.567389] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15715.592263] LNet: Added LNI 192.168.202.108@tcp49 [8/256/0/180] [15722.406871] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15722.441798] LNet: Added LNI 192.168.202.108@tcp50 [8/256/0/180] [15739.884739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15739.896751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15739.916503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15741.923243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15741.925041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15741.941196] Lustre: Skipped 1 previous similar message [15741.951151] Lustre: Skipped 2 previous similar messages [15747.044510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15747.052496] Lustre: Skipped 2 previous similar messages [15752.163448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15752.166288] Lustre: Skipped 1 previous similar message [15752.166400] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [15752.305733] Lustre: server umount lustre-MDT0000 complete [15754.410588] LustreError: 444346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890148 with bad export cookie 13861604382492532574 [15754.416618] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15754.420890] LustreError: 444346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15754.622487] Lustre: server umount lustre-MDT0001 complete [15766.318886] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [15771.665386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15773.408120] Lustre: 443627:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774890151/real 1774890151] req@00000000dd420ca9 x1861106926750080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774890167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15773.429269] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15775.712140] Lustre: 443629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774890153/real 1774890153] req@00000000c304dd61 x1861106926750208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774890169 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15778.784520] Lustre: 443629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774890156/real 1774890156] req@00000000831e6584 x1861106926750336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774890172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15779.805089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15780.832145] Lustre: 443626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774890158/real 1774890158] req@000000003294acc4 x1861106926750592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774890174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15782.789569] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15782.891418] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15785.954225] Lustre: 443626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1774890163/real 1774890163] req@00000000f42f75cb x1861106926750720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1774890179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15788.521846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15788.704974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15791.353994] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15794.152267] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15794.974731] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15797.467691] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15810.534125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15810.539806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15810.545594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15814.628769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15814.635580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15816.578526] Lustre: server umount lustre-OST0000 complete [15819.747696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15819.751746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15819.766049] Lustre: Skipped 3 previous similar messages [15825.061195] Lustre: server umount lustre-MDT0000 complete [15827.379471] LustreError: 445141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890221 with bad export cookie 13861604382492534975 [15827.383090] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15827.388214] LustreError: 445141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15827.579259] Lustre: server umount lustre-MDT0001 complete [15834.055248] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [15835.730656] Key type lgssc unregistered [15835.985451] LNet: 472098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15835.988380] LNetError: 472098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15835.993065] LNet: Removed LNI 192.168.202.108@tcp [15835.994773] LNet: Removed LNI 192.168.202.108@tcp10 [15835.996479] LNet: Removed LNI 192.168.202.108@tcp11 [15835.998665] LNet: Removed LNI 192.168.202.108@tcp12 [15836.000799] LNet: Removed LNI 192.168.202.108@tcp13 [15836.002338] LNet: Removed LNI 192.168.202.108@tcp14 [15836.004132] LNet: Removed LNI 192.168.202.108@tcp15 [15836.005598] LNet: Removed LNI 192.168.202.108@tcp16 [15836.007937] LNet: Removed LNI 192.168.202.108@tcp17 [15836.010238] LNet: Removed LNI 192.168.202.108@tcp18 [15836.011845] LNet: Removed LNI 192.168.202.108@tcp19 [15836.013343] LNet: Removed LNI 192.168.202.108@tcp20 [15836.015122] LNet: Removed LNI 192.168.202.108@tcp21 [15836.016702] LNet: Removed LNI 192.168.202.108@tcp22 [15836.018194] LNet: Removed LNI 192.168.202.108@tcp23 [15836.019807] LNet: Removed LNI 192.168.202.108@tcp24 [15836.021744] LNet: Removed LNI 192.168.202.108@tcp25 [15836.023995] LNet: Removed LNI 192.168.202.108@tcp26 [15836.026482] LNet: Removed LNI 192.168.202.108@tcp27 [15836.028321] LNet: Removed LNI 192.168.202.108@tcp28 [15836.030240] LNet: Removed LNI 192.168.202.108@tcp29 [15836.031728] LNet: Removed LNI 192.168.202.108@tcp30 [15836.033334] LNet: Removed LNI 192.168.202.108@tcp31 [15836.035071] LNet: Removed LNI 192.168.202.108@tcp32 [15836.036978] LNet: Removed LNI 192.168.202.108@tcp33 [15836.038578] LNet: Removed LNI 192.168.202.108@tcp34 [15836.040772] LNet: Removed LNI 192.168.202.108@tcp35 [15836.042780] LNet: Removed LNI 192.168.202.108@tcp36 [15836.044149] LNet: Removed LNI 192.168.202.108@tcp37 [15836.045884] LNet: Removed LNI 192.168.202.108@tcp38 [15836.047657] LNet: Removed LNI 192.168.202.108@tcp39 [15836.049727] LNet: Removed LNI 192.168.202.108@tcp40 [15836.052151] LNet: Removed LNI 192.168.202.108@tcp41 [15836.054087] LNet: Removed LNI 192.168.202.108@tcp42 [15836.055956] LNet: Removed LNI 192.168.202.108@tcp43 [15836.057171] LNet: Removed LNI 192.168.202.108@tcp44 [15836.058614] LNet: Removed LNI 192.168.202.108@tcp45 [15836.061204] LNet: Removed LNI 192.168.202.108@tcp46 [15836.062697] LNet: Removed LNI 192.168.202.108@tcp47 [15836.064528] LNet: Removed LNI 192.168.202.108@tcp48 [15836.066279] LNet: Removed LNI 192.168.202.108@tcp49 [15836.077125] LNet: Removed LNI 192.168.202.108@tcp50 [15836.564990] Key type .llcrypt unregistered [15836.567063] Key type ._llcrypt unregistered [15849.552057] Key type ._llcrypt registered [15849.555133] Key type .llcrypt registered [15849.626191] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [15850.192664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15850.320412] alg: No test for adler32 (adler32-zlib) [15851.308225] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [15851.464614] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [15853.080171] Key type lgssc registered [15853.869060] Lustre: Echo OBD driver; http://www.lustre.org/ [15859.393554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15859.409740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15860.777544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15863.574150] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15868.747807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15868.990429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15871.514284] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15875.358163] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15877.933243] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15882.355076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15882.674927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15885.751605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [15886.773439] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15890.019723] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15900.859111] loop0: detected capacity change from 0 to 204800000 [15900.953658] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15900.959829] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15900.973629] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15904.234591] loop0: detected capacity change from 0 to 204800000 [15904.371333] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15904.377977] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15904.384824] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15907.187489] loop0: detected capacity change from 0 to 204800000 [15910.003396] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15910.064319] Lustre: fs15246-OST0000: new disk, initializing [15910.066713] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15910.100028] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15913.100271] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15919.179818] loop1: detected capacity change from 0 to 204800000 [15922.544777] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15922.588447] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15922.614786] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15922.656921] Lustre: fs15246-MDT0000: new disk, initializing [15922.743297] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15922.755811] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15922.826312] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15923.875838] Lustre: 477682:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000bc3999e1 x1861107313577600/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1774890333 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15924.876697] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [15932.974940] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15932.982226] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15933.013249] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15942.120412] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15942.135147] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15943.651737] Lustre: server umount fs15246-MDT0000 complete [15947.238379] LustreError: 474221:0:(ldlm_lib.c:1178:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15947.246740] LustreError: 474221:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [15948.023177] Lustre: server umount fs15246-OST0000 complete [15955.427136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15955.434426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15955.443750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15957.474129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15957.474238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15962.596552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15962.601273] Lustre: Skipped 2 previous similar messages [15967.719672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15967.724445] Lustre: Skipped 1 previous similar message [15968.227041] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15968.301595] Lustre: server umount lustre-OST0000 complete [15971.297159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15971.300838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15976.454405] Lustre: server umount lustre-MDT0000 complete [15977.954169] LustreError: 473323:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15977.964047] LustreError: 473323:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [15978.368512] LustreError: 477676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890372 with bad export cookie 8754700237567352860 [15978.372323] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15978.373735] LustreError: 477676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15978.516326] Lustre: server umount lustre-MDT0001 complete [15983.881821] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [15985.231844] Key type lgssc unregistered [15985.376399] LNet: 480743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15985.380544] LNetError: 480743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15985.395462] LNet: Removed LNI 192.168.202.108@tcp [15985.825345] Key type .llcrypt unregistered [15985.827283] Key type ._llcrypt unregistered [15995.700423] Key type ._llcrypt registered [15995.703343] Key type .llcrypt registered [15995.754204] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [15996.277896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15996.313791] alg: No test for adler32 (adler32-zlib) [15997.214659] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [15997.340339] LNet: Added LNI 192.168.202.108@tcp [8/256/0/180] [15998.961515] Key type lgssc registered [15999.514778] Lustre: Echo OBD driver; http://www.lustre.org/ [16019.593605] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [16024.187849] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [16044.376765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16044.401940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16045.547092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16045.578722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16045.642581] Lustre: lustre-MDT0000: new disk, initializing [16045.693232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16045.705947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16047.814752] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [16053.603140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16053.667387] Lustre: 485133:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16053.701604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16053.705634] Lustre: Skipped 1 previous similar message [16053.755936] Lustre: lustre-MDT0001: new disk, initializing [16053.799960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16053.828439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16053.836422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16055.773116] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [16060.190905] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16061.964445] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16065.530477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16065.680218] Lustre: lustre-OST0000: new disk, initializing [16065.685685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16065.749923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16067.196205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16067.204930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16067.264910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16068.554292] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_default_debug -1 all [16072.953499] Lustre: DEBUG MARKER: oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16074.850442] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16074.940362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16076.918371] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16077.022792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16082.914489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16082.914889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16082.918067] LustreError: Skipped 1 previous similar message [16082.922907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16082.927423] Lustre: Skipped 1 previous similar message [16088.035619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16088.038023] Lustre: Skipped 1 previous similar message [16092.129131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16092.205440] Lustre: server umount lustre-OST0000 complete [16094.689721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16094.695609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16094.700390] Lustre: Skipped 1 previous similar message [16094.703992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16098.273487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16098.273730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16100.224831] Lustre: server umount lustre-MDT0000 complete [16102.067405] LustreError: 485128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1774890495 with bad export cookie 2911610699023073645 [16102.067975] LustreError: MGC192.168.202.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16102.074594] LustreError: 485128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16102.220515] Lustre: server umount lustre-MDT0001 complete [16113.541118] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing set_hostid [16118.243846] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing load_modules_local [16134.496796] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15765 sec ========= 13:08:46 (1774890526) [16135.526770] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:08:47 (1774890527) === [16137.170187] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:08:49 (1774890529) === [16156.831400] Lustre: DEBUG MARKER: oleg208-server.virtnet: executing unload_modules_local [16158.157862] Key type lgssc unregistered [16158.312593] LNet: 493306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16158.315702] LNetError: 493306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16158.324356] LNet: Removed LNI 192.168.202.108@tcp [16158.661205] Key type .llcrypt unregistered [16158.662560] Key type ._llcrypt unregistered