[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 731947099 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002391] x2apic enabled [ 0.003014] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009144] LSM: Security Framework initializing [ 0.010073] Yama: becoming mindful. [ 0.011055] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.021533] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026190] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029138] *** VALIDATE tmpfs *** [ 0.030493] *** VALIDATE proc *** [ 0.032263] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034299] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035181] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037035] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041677] debug: unmapping init [mem 0xffffffffb1459000-0xffffffffb1460fff] [ 0.044296] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045872] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046091] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049019] ... value mask: 0000ffffffffffff [ 0.050019] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053414] rcu: Hierarchical SRCU implementation. [ 0.055711] smp: Bringing up secondary CPUs ... [ 0.056694] x86: Booting SMP configuration: [ 0.057039] .... node #0, CPUs: #1 #2 #3 [ 0.064014] smp: Brought up 1 node, 4 CPUs [ 0.066039] smpboot: Max logical packages: 1 [ 0.067022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128370] node 0 deferred pages initialised in 58ms [ 0.132487] devtmpfs: initialized [ 0.133365] x86/mm: Memory block size: 128MB [ 0.135000] gcov: version magic: 0x41383552 [ 0.146755] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159699] pinctrl core: initialized pinctrl subsystem [ 0.163428] [ 0.164265] ************************************************************* [ 0.168019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173018] ** ** [ 0.176015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181019] ** ** [ 0.184016] ** This means that this kernel is built to expose internal ** [ 0.186033] ** IOMMU data structures, which may compromise security on ** [ 0.189020] ** your system. ** [ 0.194018] ** ** [ 0.200017] ** If you see this message and you are not debugging the ** [ 0.204018] ** kernel, report this immediately to your vendor! ** [ 0.210018] ** ** [ 0.217018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.222016] ************************************************************* [ 0.229000] NET: Registered protocol family 16 [ 0.234100] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.239081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.244081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.250284] cpuidle: using governor menu [ 0.251537] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.255771] PCI: Using configuration type 1 for base access [ 0.259252] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.269064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.270045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.272684] cryptd: max_cpu_qlen set to 1000 [ 0.275144] ACPI: Added _OSI(Module Device) [ 0.276015] ACPI: Added _OSI(Processor Device) [ 0.279020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.289040] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.297971] ACPI: Interpreter enabled [ 0.300105] ACPI: PM: (supports S0 S3 S4 S5) [ 0.303017] ACPI: Using IOAPIC for interrupt routing [ 0.307109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.313524] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.330000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.335060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.340024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.346093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.356858] acpiphp: Slot [2] registered [ 0.359283] acpiphp: Slot [5] registered [ 0.362143] acpiphp: Slot [6] registered [ 0.364290] acpiphp: Slot [7] registered [ 0.367284] acpiphp: Slot [8] registered [ 0.370369] acpiphp: Slot [9] registered [ 0.375203] acpiphp: Slot [10] registered [ 0.380012] acpiphp: Slot [3] registered [ 0.382207] acpiphp: Slot [4] registered [ 0.384119] acpiphp: Slot [11] registered [ 0.385000] acpiphp: Slot [12] registered [ 0.385000] acpiphp: Slot [13] registered [ 0.386130] acpiphp: Slot [14] registered [ 0.388135] acpiphp: Slot [15] registered [ 0.389128] acpiphp: Slot [16] registered [ 0.391147] acpiphp: Slot [17] registered [ 0.392202] acpiphp: Slot [18] registered [ 0.394128] acpiphp: Slot [19] registered [ 0.396120] acpiphp: Slot [20] registered [ 0.398143] acpiphp: Slot [21] registered [ 0.399132] acpiphp: Slot [22] registered [ 0.401115] acpiphp: Slot [23] registered [ 0.402124] acpiphp: Slot [24] registered [ 0.404161] acpiphp: Slot [25] registered [ 0.406129] acpiphp: Slot [26] registered [ 0.408192] acpiphp: Slot [27] registered [ 0.409116] acpiphp: Slot [28] registered [ 0.411132] acpiphp: Slot [29] registered [ 0.413138] acpiphp: Slot [30] registered [ 0.415165] acpiphp: Slot [31] registered [ 0.416067] PCI host bridge to bus 0000:00 [ 0.418020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.421027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.424028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.427055] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.430083] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.433033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.435187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.439000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.442375] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.452825] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.461016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.463020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.467020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.471024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.473678] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.476881] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.480048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.483884] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.489018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.501017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.506016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.515381] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.524018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.533019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.552020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.597075] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.612023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.624017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.655017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.674000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.687000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.696018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.725016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.738043] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.748017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.767019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.805026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.826359] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.836019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.849036] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.868022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.881980] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.893021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.903024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.933019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.954244] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.977797] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.981756] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.985767] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.988382] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.995197] iommu: Default domain type: Passthrough [ 0.998027] SCSI subsystem initialized [ 1.001458] ACPI: bus type USB registered [ 1.002220] usbcore: registered new interface driver usbfs [ 1.006172] usbcore: registered new interface driver hub [ 1.009175] usbcore: registered new device driver usb [ 1.013450] pps_core: LinuxPPS API ver. 1 registered [ 1.017014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.022271] PTP clock support registered [ 1.026079] EDAC MC: Ver: 3.0.0 [ 1.033123] PCI: Using ACPI for IRQ routing [ 1.039480] NetLabel: Initializing [ 1.041015] NetLabel: domain hash size = 128 [ 1.045015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.051102] NetLabel: unlabeled traffic allowed by default [ 1.058285] vgaarb: loaded [ 1.063308] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.067020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.073447] clocksource: Switched to clocksource kvm-clock [ 1.426147] VFS: Disk quotas dquot_6.6.0 [ 1.428213] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.431340] *** VALIDATE ramfs *** [ 1.432782] *** VALIDATE hugetlbfs *** [ 1.434723] pnp: PnP ACPI init [ 1.437593] pnp: PnP ACPI: found 6 devices [ 1.468871] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.475752] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.480085] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.484166] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.489648] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.494336] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.502390] NET: Registered protocol family 2 [ 1.507807] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.513377] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.517456] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.523428] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.527518] TCP: Hash tables configured (established 65536 bind 65536) [ 1.530943] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.534587] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.537853] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.541865] NET: Registered protocol family 1 [ 1.550923] RPC: Registered named UNIX socket transport module. [ 1.557730] RPC: Registered udp transport module. [ 1.566740] RPC: Registered tcp transport module. [ 1.569567] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.600658] NET: Registered protocol family 44 [ 1.609436] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.612051] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.614591] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.617419] PCI: CLS 0 bytes, default 64 [ 1.619492] Unpacking initramfs... [ 3.453417] debug: unmapping init [mem 0xffff9f79bcc54000-0xffff9f79bffbffff] [ 3.460280] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.463238] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.466921] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.704402] Initialise system trusted keyrings [ 4.742511] Key type blacklist registered [ 4.749468] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.772254] zbud: loaded [ 4.776141] *** VALIDATE nfs *** [ 4.777737] *** VALIDATE nfs4 *** [ 4.779639] pstore: using deflate compression [ 4.786552] Platform Keyring initialized [ 4.919867] NET: Registered protocol family 38 [ 4.923173] Key type asymmetric registered [ 4.925233] Asymmetric key parser 'x509' registered [ 4.927645] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.931596] io scheduler mq-deadline registered [ 4.933229] io scheduler kyber registered [ 4.935041] io scheduler bfq registered [ 4.941577] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.946883] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.950639] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.954519] ACPI: Power Button [PWRF] [ 4.961907] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.971920] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.990393] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.998832] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.115287] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.151243] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.182106] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.186816] Non-volatile memory driver v1.3 [ 5.189042] Linux agpgart interface v0.103 [ 5.260733] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.264211] vda: detected capacity change from 0 to 74702848 [ 5.320434] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.324067] vdb: detected capacity change from 0 to 1073741824 [ 5.345107] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.348978] vdc: detected capacity change from 0 to 2621440000 [ 5.400407] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.403646] vdd: detected capacity change from 0 to 2621440000 [ 5.421285] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.424666] vde: detected capacity change from 0 to 4294967296 [ 5.442445] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.445689] vdf: detected capacity change from 0 to 4294967296 [ 5.451908] libphy: Fixed MDIO Bus: probed [ 5.457871] usbcore: registered new interface driver usbserial_generic [ 5.460459] usbserial: USB Serial support registered for generic [ 5.462964] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.467577] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.469740] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.472727] mousedev: PS/2 mouse device common for all mice [ 5.476400] rtc_cmos 00:05: RTC can wake from S4 [ 5.483876] rtc_cmos 00:05: registered as rtc0 [ 5.488691] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.491366] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.491422] intel_pstate: CPU model not supported [ 5.496603] hid: raw HID events driver (C) Jiri Kosina [ 5.507993] usbcore: registered new interface driver usbhid [ 5.510595] usbhid: USB HID core driver [ 5.513543] drop_monitor: Initializing network drop monitor service [ 5.517021] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.521108] Initializing XFRM netlink socket [ 5.521663] NET: Registered protocol family 10 [ 5.523851] Segment Routing with IPv6 [ 5.530431] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.533687] NET: Registered protocol family 17 [ 5.542626] mpls_gso: MPLS GSO support [ 5.549488] RAS: Correctable Errors collector initialized. [ 5.552226] AVX version of gcm_enc/dec engaged. [ 5.554203] AES CTR mode by8 optimization enabled [ 5.707241] sched_clock: Marking stable (5707104478, 0)->(6937951555, -1230847077) [ 5.712692] registered taskstats version 1 [ 5.716409] Loading compiled-in X.509 certificates [ 5.719296] zswap: loaded using pool lzo/zbud [ 5.789300] Key type big_key registered [ 5.875924] Key type encrypted registered [ 5.878434] ima: No TPM chip found, activating TPM-bypass! [ 5.881532] ima: Allocated hash algorithm: sha1 [ 5.884245] ima: No architecture policies found [ 5.886960] evm: Initialising EVM extended attributes: [ 5.889927] evm: security.selinux [ 5.891887] evm: security.ima [ 5.893898] evm: security.capability [ 5.896415] evm: HMAC attrs: 0x1 [ 5.901290] rtc_cmos 00:05: setting system clock to 2026-08-09 20:54:54 UTC (1786308894) [ 5.914962] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 5.920359] debug: unmapping init [mem 0xffffffffb1182000-0xffffffffb1458fff] [ 5.928143] Write protecting the kernel read-only data: 28672k [ 5.934292] debug: unmapping init [mem 0xffffffffaf803000-0xffffffffaf9fffff] [ 5.939338] debug: unmapping init [mem 0xffffffffb0114000-0xffffffffb01fffff] [ 5.998781] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.010556] systemd[1]: Detected virtualization kvm. [ 6.013199] systemd[1]: Detected architecture x86-64. [ 6.015976] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.095442] systemd[1]: No hostname configured. [ 6.097814] systemd[1]: Set hostname to . [ 6.100713] random: systemd: uninitialized urandom read (16 bytes read) [ 6.104219] systemd[1]: Initializing machine ID from random generator. [ 6.234045] random: ln: uninitialized urandom read (6 bytes read) [ 6.632644] random: systemd: uninitialized urandom read (16 bytes read) [ 6.653655] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.676541] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.694849] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.351543] device-mapper: uevent: version 1.0.3 [ 8.360762] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [[ 9.972933] random: fast init done  OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.145499] virtio_net virtio0 ens2: renamed from eth0 [ 10.614695] scsi host0: ata_piix [ 10.669702] scsi host1: ata_piix [ 10.671902] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.676344] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.460593] random: crng init done [ 15.463637] random: 7 urandom warning(s) missed due to ratelimiting [ 17.038405] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.296200] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.747794] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.291061] SELinux: Disabled at runtime. [ 21.363220] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.371557] systemd[1]: Detected virtualization kvm. [ 21.372991] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.542745] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.549738] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.560282] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.566222] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.572130] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.594132] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.604960] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 23.154715] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.911702] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.763196] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.946050] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.311687] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.492353] EDAC sbridge: Ver: 1.1.2 [ 28.620983] Key type dns_resolver registered [ 29.135890] NFS: Registering the id_resolver key type [ 29.138669] Key type id_resolver registered [ 29.141394] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg312-server login: [ 81.119237] libcfs: loading out-of-tree module taints kernel. [ 81.179596] Key type ._llcrypt registered [ 81.188602] Key type .llcrypt registered [ 81.302746] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 97.156701] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 98.738869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.760044] alg: No test for adler32 (adler32-zlib) [ 100.305852] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 101.283057] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 103.079180] Key type lgssc registered [ 105.134851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.263384] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 153.638839] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 166.051653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.074709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.310347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.431229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.543901] Lustre: lustre-MDT0000: new disk, initializing [ 167.631870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.666385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.704391] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.162163] hrtimer: interrupt took 2176032 ns [ 185.690130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.786218] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.819142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.827391] Lustre: Skipped 1 previous similar message [ 185.904832] Lustre: lustre-MDT0001: new disk, initializing [ 185.974993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.034210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.046584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.009670] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.617572] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 205.629124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.889347] Lustre: lustre-OST0000: new disk, initializing [ 205.892531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.897326] Lustre: 8453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.958953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.547411] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.568169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.578573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.671249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.780798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.869839] Lustre: lustre-OST0001: new disk, initializing [ 226.873558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.878655] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.947885] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.941098] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.526178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 234.540897] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 234.677992] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.902698] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.986626] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 255.406995] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.207731] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing yml_node [ 264.505312] Lustre: DEBUG MARKER: Client: 2.17.56.55 [ 267.763867] Lustre: DEBUG MARKER: MDS: 2.17.56.55 [ 270.644533] Lustre: DEBUG MARKER: OSS: 2.17.56.55 [ 272.259492] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 9 16:59:18 EDT 2026 [ 289.388343] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 290.625928] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.657032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.662757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.675984] Lustre: Skipped 3 previous similar messages [ 306.681450] Lustre: Skipped 2 previous similar messages [ 311.776749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.789737] Lustre: Skipped 3 previous similar messages [ 312.379298] Lustre: server umount lustre-MDT0000 complete [ 315.652102] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309204 with bad export cookie 12019531847937543964 [ 315.655994] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.705572] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.898166] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.898361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.948852] Lustre: Skipped 1 previous similar message [ 316.973413] Lustre: Skipped 1 previous similar message [ 320.938127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.941950] Lustre: Skipped 1 previous similar message [ 322.201150] Lustre: server umount lustre-MDT0001 complete [ 326.749344] Lustre: server umount lustre-OST0000 complete [ 330.014987] Lustre: server umount lustre-OST0001 complete [ 336.659439] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 344.718104] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 380.071887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.236694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 380.259305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 380.328414] Lustre: lustre-MDT0000: new disk, initializing [ 380.380788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.397795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.229501] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 394.035176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.138825] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 394.185828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 394.189340] Lustre: Skipped 1 previous similar message [ 394.306130] Lustre: lustre-MDT0001: new disk, initializing [ 394.402406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 394.444591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 394.461270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 398.283790] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 406.604203] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 412.830642] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 420.451567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 420.879411] Lustre: lustre-OST0000: new disk, initializing [ 420.882430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 420.887990] Lustre: 17059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 420.968724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.784992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 422.805313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 423.015243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 427.138127] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 436.267541] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 441.321339] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 441.618975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 445.539303] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 445.751512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 448.482737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.490368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 448.491208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 448.497180] Lustre: Skipped 1 previous similar message [ 453.492826] Lustre: server umount lustre-OST0000 complete [ 458.721518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.731134] Lustre: Skipped 1 previous similar message [ 458.732580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 458.738507] Lustre: Skipped 2 previous similar messages [ 463.712487] Lustre: server umount lustre-MDT0000 complete [ 463.852413] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 463.870775] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 466.953285] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309355 with bad export cookie 12019531847937545504 [ 466.958418] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 466.965429] LustreError: 15079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 467.245910] Lustre: server umount lustre-MDT0001 complete [ 470.603569] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:02:37 (1786309357) [ 473.133374] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 475.234847] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:02:41 (1786309361) [ 480.746633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.106875] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.173507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.937279] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 486.425104] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.598830] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.980854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.866248] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 506.928060] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 513.301531] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 520.958563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.935509] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 535.293280] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 544.742596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.749248] LustreError: Skipped 1 previous similar message [ 544.752506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.754544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.762671] Lustre: Skipped 1 previous similar message [ 544.778322] Lustre: Skipped 1 previous similar message [ 559.071555] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 559.177990] Lustre: server umount lustre-OST0000 complete [ 565.219967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.233082] Lustre: Skipped 1 previous similar message [ 568.323954] Lustre: server umount lustre-MDT0000 complete [ 570.354742] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.390923] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 572.193748] LustreError: 21960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309460 with bad export cookie 12019531847937546386 [ 572.197119] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.203678] LustreError: 21960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 572.889683] Lustre: server umount lustre-MDT0001 complete [ 582.697466] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 585.530312] Key type lgssc unregistered [ 585.744754] LNet: 23004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.752710] LNetError: 23004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.768854] LNet: Removed LNI 192.168.203.112@tcp [ 586.586305] Key type .llcrypt unregistered [ 586.594652] Key type ._llcrypt unregistered [ 607.969500] Key type ._llcrypt registered [ 607.982935] Key type .llcrypt registered [ 608.134462] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 609.429321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 609.547043] alg: No test for adler32 (adler32-zlib) [ 610.725598] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 611.212264] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 612.911172] Key type lgssc registered [ 614.039423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.995317] loop: module loaded [ 622.108404] loop0: detected capacity change from 0 to 204800000 [ 622.387121] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.402149] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.584954] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.774411] loop0: detected capacity change from 0 to 204800000 [ 627.993404] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.999854] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.007368] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.270350] loop0: detected capacity change from 0 to 204800000 [ 638.533549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 638.550628] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.775597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 639.814634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 639.874293] Lustre: lustre-MDT0000: new disk, initializing [ 639.958911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 640.044938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 643.922668] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 651.862377] loop1: detected capacity change from 0 to 204800000 [ 657.229325] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 657.443476] Lustre: lustre-OST07c6: new disk, initializing [ 657.446495] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 657.451210] Lustre: Skipped 1 previous similar message [ 657.456054] Lustre: 26190:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 657.524905] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 662.975369] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 665.153589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 665.161950] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 665.217904] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 667.970557] Lustre: 25177:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 667.985574] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 667.991109] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 667.995942] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 668.000935] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 668.005801] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 681.589752] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 681.602295] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 681.608772] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 681.617152] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 681.623597] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 681.630283] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.529224] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 686.545885] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 686.563046] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 686.572775] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.584114] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.589773] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.595162] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 686.601713] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.607456] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.613257] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.618620] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.624781] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.803588] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 689.810829] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 689.817765] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 689.824272] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.833712] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 689.841896] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.848874] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 689.857247] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.863577] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 689.870812] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.877167] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.884678] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.294508] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 694.301159] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 694.306164] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 694.312200] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.318042] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 694.328639] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.335367] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 694.342189] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.346615] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 694.350738] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.354687] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 694.358677] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.124923] Lustre: 25177:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 705.132103] Lustre: 25177:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 705.136265] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 705.139743] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.143414] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 705.148433] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.152947] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 705.157478] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.161298] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 705.165704] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 705.169075] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 705.172792] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 752.098265] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.099928] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 752.108222] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 757.217432] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 758.263905] Lustre: server umount lustre-OST07c6 complete [ 767.280710] Lustre: server umount lustre-MDT0000 complete [ 783.103404] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 786.299212] Key type lgssc unregistered [ 786.522727] LNet: 28750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.536893] LNetError: 28750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.577422] LNet: Removed LNI 192.168.203.112@tcp [ 787.348599] Key type .llcrypt unregistered [ 787.350766] Key type ._llcrypt unregistered [ 805.402702] Key type ._llcrypt registered [ 805.406449] Key type .llcrypt registered [ 805.512991] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 806.324769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.369080] alg: No test for adler32 (adler32-zlib) [ 807.503643] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 807.667802] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 809.295176] Key type lgssc registered [ 810.274815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 817.799153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 817.815629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.225976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.888231] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 829.427753] LustreError: 30015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 829.457053] LustreError: 30015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 831.485744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.768341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 836.224669] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 844.591438] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 849.269288] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 855.736825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 855.965218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.533645] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 865.350883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 869.572200] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 885.731824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 885.732818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.745262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.747500] Lustre: Skipped 1 previous similar message [ 885.758047] Lustre: Skipped 1 previous similar message [ 888.896270] Lustre: server umount lustre-OST0000 complete [ 893.408171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 893.418518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.441787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.970124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.970714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.992908] Lustre: Skipped 1 previous similar message [ 898.888650] Lustre: server umount lustre-MDT0000 complete [ 901.109094] LustreError: 30011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.141360] LustreError: 30011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 902.730066] LustreError: 29997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309791 with bad export cookie 12478850873563407922 [ 902.741158] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.748200] LustreError: 29997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 903.034595] Lustre: server umount lustre-MDT0001 complete [ 913.605145] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 916.528954] Key type lgssc unregistered [ 916.808669] LNet: 33754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.813702] LNetError: 33754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.826757] LNet: Removed LNI 192.168.203.112@tcp [ 917.502513] Key type .llcrypt unregistered [ 917.504539] Key type ._llcrypt unregistered [ 933.441314] Key type ._llcrypt registered [ 933.442790] Key type .llcrypt registered [ 933.584992] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 934.467912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 934.781770] alg: No test for adler32 (adler32-zlib) [ 936.643328] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 936.871928] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 938.511245] Key type lgssc registered [ 939.709674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 947.044657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 947.057608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.423828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.374809] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 957.924464] LustreError: 34998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.945803] LustreError: 34998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 960.403257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.710416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 965.380512] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 972.564858] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.114907] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 982.894831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.157127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.766223] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 993.403898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 996.299811] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1007.077111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1007.095172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.118153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.134692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.136672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.152515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.189169] Lustre: Skipped 1 previous similar message [ 1011.388234] Lustre: server umount lustre-OST0000 complete [ 1017.312251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1017.321180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.334247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.362187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.365288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.384473] Lustre: Skipped 1 previous similar message [ 1020.653624] Lustre: server umount lustre-MDT0000 complete [ 1024.482398] LustreError: 35726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.507916] LustreError: 35726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1024.774346] LustreError: 34979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309913 with bad export cookie 2576131929675709118 [ 1024.791915] LustreError: 34979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1024.799046] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1025.091242] Lustre: server umount lustre-MDT0001 complete [ 1034.505710] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1036.379850] Key type lgssc unregistered [ 1036.631234] LNet: 38531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.637616] LNetError: 38531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.672434] LNet: Removed LNI 192.168.203.112@tcp [ 1038.456625] Key type .llcrypt unregistered [ 1038.458416] Key type ._llcrypt unregistered [ 1061.461232] Key type ._llcrypt registered [ 1061.463228] Key type .llcrypt registered [ 1061.539506] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1062.902273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1062.944953] alg: No test for adler32 (adler32-zlib) [ 1064.282826] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1064.511643] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1066.183336] Key type lgssc registered [ 1067.624644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1074.554150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1074.566853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.102863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.405414] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1085.500397] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.532802] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1088.319388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.544396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1092.435777] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1100.246878] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1105.005913] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1110.901966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.147575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.844624] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1121.396510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1124.102899] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1132.002177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.006850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.016084] Lustre: Skipped 2 previous similar messages [ 1132.023955] Lustre: Skipped 2 previous similar messages [ 1132.610868] LustreError: 39945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.623739] LustreError: 42526:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000003f50c671) refcount nonzero (0) after lock cleanup; forcing cleanup. [ 1132.677886] Lustre: server umount lustre-MDT0000 complete [ 1135.010104] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.020954] LustreError: 39948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1135.417347] LustreError: 39929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310024 with bad export cookie 13316480953322548740 [ 1135.422585] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.425764] LustreError: 39929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1137.707265] LustreError: 39943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.732817] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.12@tcp (stopping) [ 1140.197350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.218066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1141.681651] Lustre: server umount lustre-MDT0001 complete [ 1161.619838] Lustre: server umount lustre-OST0000 complete [ 1174.382545] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1176.611754] Key type lgssc unregistered [ 1176.860787] LNet: 43812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.865513] LNetError: 43812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.877225] LNet: Removed LNI 192.168.203.112@tcp [ 1177.597907] Key type .llcrypt unregistered [ 1177.601694] Key type ._llcrypt unregistered [ 1198.330939] Key type ._llcrypt registered [ 1198.332693] Key type .llcrypt registered [ 1198.409285] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1200.182398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1200.249179] alg: No test for adler32 (adler32-zlib) [ 1201.284662] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1201.553891] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1203.199264] Key type lgssc registered [ 1204.235811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1210.837291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1210.853398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.286085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.705303] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1221.607547] LustreError: 45208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.619537] LustreError: 45208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1224.274330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.554455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1229.020460] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1235.775372] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1240.752197] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1246.880166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1247.416097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1252.181393] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1256.646576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1257.928923] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1264.724229] Lustre: lustre-OST0000: Not available for connect from 192.168.203.12@tcp (stopping) [ 1265.120665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1265.129810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.189298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.190423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1267.215479] Lustre: Skipped 1 previous similar message [ 1269.181951] Lustre: server umount lustre-OST0000 complete [ 1286.119632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.129759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.139674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1291.745888] LustreError: 46228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.758411] LustreError: 46228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1291.785241] Lustre: server umount lustre-MDT0000 complete [ 1295.014919] LustreError: 45190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310183 with bad export cookie 16954715002327802645 [ 1295.028719] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1301.356500] Lustre: server umount lustre-MDT0001 complete [ 1312.479954] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1314.584433] Key type lgssc unregistered [ 1314.976390] LNet: 48916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.984245] LNetError: 48916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.996863] LNet: Removed LNI 192.168.203.112@tcp [ 1315.727981] Key type .llcrypt unregistered [ 1315.730422] Key type ._llcrypt unregistered [ 1338.081717] Key type ._llcrypt registered [ 1338.083530] Key type .llcrypt registered [ 1338.170734] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1339.408474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1339.750164] alg: No test for adler32 (adler32-zlib) [ 1340.881796] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1341.121056] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1342.911408] Key type lgssc registered [ 1344.109309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1350.731632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1350.754851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.085375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.996758] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1362.406694] LustreError: 50313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.443365] LustreError: 50313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1364.459682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.723912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.466243] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1376.387654] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1381.834236] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1388.724257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.018629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.557525] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1398.332525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1402.459967] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1408.558417] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1413.147631] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1425.601840] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-0917h18m31s [ 1426.855833] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1428.962088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.964291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.972102] Lustre: Skipped 2 previous similar messages [ 1428.978891] Lustre: Skipped 2 previous similar messages [ 1429.989829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.999304] Lustre: Skipped 2 previous similar messages [ 1431.660057] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.12@tcp (stopping) [ 1431.666190] Lustre: Skipped 1 previous similar message [ 1434.727774] Lustre: server umount lustre-MDT0000 complete [ 1436.153603] LustreError: 50313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.190342] LustreError: 50313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1439.349926] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1442.277797] LustreError: 52834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.286808] LustreError: 52834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1446.276964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.377787] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1446.620285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.762185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1447.664728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1451.166571] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1454.107830] Lustre: DEBUG MARKER: Wait for df (36686) ... [ 1456.361774] Lustre: DEBUG MARKER: done [ 1461.730103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1461.744120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.752969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1462.244276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.479756] Lustre: server umount lustre-OST0000 complete [ 1473.506307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.513447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.513739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1473.530985] Lustre: Skipped 2 previous similar messages [ 1473.545264] Lustre: Skipped 3 previous similar messages [ 1484.255431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1484.518529] Lustre: server umount lustre-MDT0000 complete [ 1487.351370] LustreError: 51040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.381142] LustreError: 51040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1488.763763] LustreError: 50294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310377 with bad export cookie 7726017040837564524 [ 1488.777485] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1488.790915] LustreError: 50294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1489.103816] Lustre: server umount lustre-MDT0001 complete [ 1501.550591] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1503.990523] Key type lgssc unregistered [ 1504.219864] LNet: 55239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.235816] LNetError: 55239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.253763] LNet: Removed LNI 192.168.203.112@tcp [ 1505.836822] Key type .llcrypt unregistered [ 1505.839508] Key type ._llcrypt unregistered [ 1543.515452] Key type ._llcrypt registered [ 1543.517854] Key type .llcrypt registered [ 1543.664851] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1545.987626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1546.121130] alg: No test for adler32 (adler32-zlib) [ 1547.166905] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1547.352817] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1549.010687] Key type lgssc registered [ 1549.909831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1558.474781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1558.519847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.020696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1560.071290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1560.406211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.297634] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1574.962157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.021123] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1575.041732] Lustre: 57442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1575.258827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1580.458333] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1588.187227] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1594.009673] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1599.823732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1599.951982] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1600.172590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.875636] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1608.403182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1612.901566] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1619.568781] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-0917h21m45s [ 1621.314756] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1628.500221] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1630.739421] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1630.780944] LustreError: 57450:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1651.324629] Lustre: lustre-MDT0000: Client ef965523-69b7-4bd9-8993-e12d1c91b9d5 (at 192.168.203.12@tcp) reconnecting [ 1670.831254] LustreError: 57450:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1672.864433] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1675.080960] Lustre: DEBUG MARKER: done [ 1680.352312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1680.355058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.379383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1680.391942] Lustre: Skipped 1 previous similar message [ 1680.406052] Lustre: Skipped 1 previous similar message [ 1685.481112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1685.492871] Lustre: Skipped 1 previous similar message [ 1686.309929] Lustre: server umount lustre-OST0000 complete [ 1690.592935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.595961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.607432] Lustre: Skipped 1 previous similar message [ 1690.619735] Lustre: Skipped 1 previous similar message [ 1696.312961] Lustre: server umount lustre-MDT0000 complete [ 1699.768336] LustreError: 57437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310588 with bad export cookie 5680907111149641906 [ 1699.771176] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.774839] LustreError: 57437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1700.025633] Lustre: server umount lustre-MDT0001 complete [ 1709.640686] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 1712.275470] Key type lgssc unregistered [ 1712.531785] LNet: 61775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.568279] LNetError: 61775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.579951] LNet: Removed LNI 192.168.203.112@tcp [ 1713.512684] Key type .llcrypt unregistered [ 1713.514703] Key type ._llcrypt unregistered [ 1755.566824] Key type ._llcrypt registered [ 1755.569885] Key type .llcrypt registered [ 1755.670969] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 1757.729314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.759197] alg: No test for adler32 (adler32-zlib) [ 1758.780633] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 1758.949028] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 1760.582831] Key type lgssc registered [ 1761.613685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1764.348895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1764.372179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.587954] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1765.601263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1765.726416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.626136] Lustre: Failing over lustre-MDT0000 [ 1766.785555] Lustre: server umount lustre-MDT0000 complete [ 1773.150775] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:24:20 (1786310660) [ 1779.342087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.596931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.425749] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1790.824881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.879149] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1790.899791] Lustre: 64574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1791.130328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1794.867203] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1802.654844] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1807.924079] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1814.174550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1814.303208] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1814.457571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.963766] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1823.893475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1827.514344] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1833.248228] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1839.586345] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.588853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1839.596727] Lustre: Skipped 2 previous similar messages [ 1839.602453] Lustre: Skipped 1 previous similar message [ 1844.705801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.709768] Lustre: Skipped 3 previous similar messages [ 1845.299969] Lustre: server umount lustre-MDT0000 complete [ 1849.193518] LustreError: 64567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310737 with bad export cookie 8391658487979799301 [ 1849.212178] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1849.233889] LustreError: 64567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1849.601285] Lustre: server umount lustre-MDT0001 complete [ 1853.347283] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1858.030757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1866.208087] Lustre: 63497:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786310738/real 1786310738] req@000000006e2f74d8 x1873082473027072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786310754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1866.243808] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.407130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.579145] Lustre: 63500:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786310743/real 1786310743] req@00000000187837e2 x1873082473027200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786310759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1873.760089] LustreError: 63496:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d58b6cfb x1873082473027968/t0(0) o250->MGC192.168.203.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1874.260371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.332273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1875.310418] Lustre: 63499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786310747/real 1786310747] req@00000000db672736 x1873082473027456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786310763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1878.363609] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1880.544239] Lustre: 63497:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786310752/real 1786310752] req@0000000018289629 x1873082473027584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786310768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1881.573655] LustreError: 68343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.607933] LustreError: 68343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 1886.690033] LustreError: 68341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.704863] LustreError: 68341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1886.834459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.092277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1891.716739] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1892.350537] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1895.474920] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1898.719478] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1904.498785] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1910.762296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.818423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.854766] Lustre: Skipped 2 previous similar messages [ 1910.885551] Lustre: Skipped 2 previous similar messages [ 1912.297697] Lustre: server umount lustre-MDT0000 complete [ 1915.879231] LustreError: 68342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.888713] LustreError: 68342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1916.698873] LustreError: 64567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310805 with bad export cookie 8391658487979801576 [ 1916.707076] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.711163] LustreError: 64567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1916.960515] Lustre: server umount lustre-MDT0001 complete [ 1923.317955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1932.376050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.375510] Lustre: 63498:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786310809/real 1786310809] req@00000000ae6ab618 x1873082473054208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786310825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1937.390178] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.537144] LustreError: 63496:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ae6ab618 x1873082473055360/t0(0) o250->MGC192.168.203.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1941.836292] LustreError: 69783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.912336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.964539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1945.864722] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1945.962521] LustreError: 69783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.984975] LustreError: 69783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1949.025591] Lustre: 63497:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786310820/real 1786310820] req@0000000009944bda x1873082473054848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786310836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1949.099422] Lustre: 63497:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1954.770676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.452604] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1961.259994] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 1965.610637] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1968.256609] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1973.699608] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1980.158684] Lustre: server umount lustre-MDT0000 complete [ 1980.896951] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.898899] LustreError: 66424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.909278] Lustre: Skipped 1 previous similar message [ 1980.938772] LustreError: 66424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 1983.804109] LustreError: 64567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786310872 with bad export cookie 8391658487979802955 [ 1983.808331] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.814193] LustreError: 64567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1984.182917] Lustre: server umount lustre-MDT0001 complete [ 1991.677955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1994.812725] Lustre: DEBUG MARKER: files compared the same [ 2007.826206] Lustre: server umount lustre-OST0000 complete [ 2024.097319] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2026.261647] Key type lgssc unregistered [ 2026.470760] LNet: 74303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.476196] LNetError: 74303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.499145] LNet: Removed LNI 192.168.203.112@tcp [ 2028.402440] Key type .llcrypt unregistered [ 2028.404659] Key type ._llcrypt unregistered [ 2052.017671] Key type ._llcrypt registered [ 2052.019936] Key type .llcrypt registered [ 2052.101526] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2053.777575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.833077] alg: No test for adler32 (adler32-zlib) [ 2055.144649] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2055.364673] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2056.999210] Key type lgssc registered [ 2058.543405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2065.121987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2065.133610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.559174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.220380] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug +malloc all [ 2077.155899] LustreError: 75721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.182767] LustreError: 75721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2077.287047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.537676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2082.586302] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug +malloc all [ 2090.162808] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2096.931429] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2104.611382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2104.909790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.218449] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug +malloc all [ 2112.041415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2117.368596] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2132.447818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2132.451802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.459238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2133.989277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.989672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2134.002820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2134.010031] Lustre: Skipped 1 previous similar message [ 2135.679899] Lustre: server umount lustre-OST0000 complete [ 2144.224418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2144.224491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.231689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.234027] Lustre: Skipped 1 previous similar message [ 2144.239328] Lustre: Skipped 1 previous similar message [ 2145.735097] Lustre: server umount lustre-MDT0000 complete [ 2149.240618] LustreError: 75703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311037 with bad export cookie 3681288004352731204 [ 2149.264953] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.351718] LustreError: 75703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2149.653256] Lustre: server umount lustre-MDT0001 complete [ 2160.104996] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2162.523833] Key type lgssc unregistered [ 2162.753705] LNet: 79275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.759623] LNetError: 79275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.830795] LNet: Removed LNI 192.168.203.112@tcp [ 2164.613987] Key type .llcrypt unregistered [ 2164.616187] Key type ._llcrypt unregistered [ 2188.581817] Key type ._llcrypt registered [ 2188.583791] Key type .llcrypt registered [ 2188.670526] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2190.101060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2190.196951] alg: No test for adler32 (adler32-zlib) [ 2191.219353] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2191.449367] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2193.111156] Key type lgssc registered [ 2194.126904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2200.983094] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2200.994535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2216.480484] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2221.599894] LustreError: 80666:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [ 2248.480317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.422361] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2261.334076] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2269.244788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.305061] Lustre: *** cfs_fail_loc=706, val=0*** [ 2269.694690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.769822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2274.444274] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2274.786932] LustreError: 80692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.796308] LustreError: 80692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2279.909923] LustreError: 80691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.203525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.443680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2287.220323] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2295.303935] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2300.423233] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2303.968736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2303.972582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.980404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2304.994690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2305.003546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.400172] Lustre: server umount lustre-OST0000 complete [ 2314.209797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2314.220905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.239585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2314.243734] Lustre: Skipped 1 previous similar message [ 2320.146701] Lustre: server umount lustre-MDT0000 complete [ 2320.359443] LustreError: 81757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.380091] LustreError: 81757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2323.634955] LustreError: 80674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311212 with bad export cookie 16788617172833102937 [ 2323.635782] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.650610] LustreError: 80674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2323.925786] Lustre: server umount lustre-MDT0001 complete [ 2333.448698] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2336.188766] Key type lgssc unregistered [ 2336.455651] LNet: 84283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.469206] LNetError: 84283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.491152] LNet: Removed LNI 192.168.203.112@tcp [ 2337.732462] Key type .llcrypt unregistered [ 2337.737177] Key type ._llcrypt unregistered [ 2361.324760] Key type ._llcrypt registered [ 2361.326950] Key type .llcrypt registered [ 2361.418923] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2362.647860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.832716] alg: No test for adler32 (adler32-zlib) [ 2363.951615] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2364.242282] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2365.983160] Key type lgssc registered [ 2368.156657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.511443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.523947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.444669] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2386.856216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.187941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.896778] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2392.548232] LustreError: 86390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.563301] LustreError: 86390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2396.642737] LustreError: 86391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.661142] LustreError: 86391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2397.914194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.337079] LustreError: 86386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.353171] LustreError: 86386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2398.375100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.813753] LustreError: 86385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.823324] LustreError: 86385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2404.226323] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2408.938110] LustreError: 86387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.978521] LustreError: 86387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2411.667715] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.676045] Lustre: MGS: Client 01e275c4-cd17-444c-8b4b-bb129aee68d4 (at 0@lo) reconnecting [ 2411.687741] LustreError: 86401:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2411.877971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.938073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2415.998129] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2428.386755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.388325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2428.389664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2428.398613] Lustre: Skipped 1 previous similar message [ 2433.585489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2433.596818] Lustre: Skipped 2 previous similar messages [ 2434.170722] Lustre: server umount lustre-OST0000 complete [ 2438.627774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.631237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2438.631767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2438.648827] Lustre: Skipped 1 previous similar message [ 2443.241810] Lustre: server umount lustre-MDT0000 complete [ 2443.746317] LustreError: 87929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.766850] LustreError: 87929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2446.211972] Lustre: server umount lustre-MDT0001 complete [ 2449.031237] Lustre: server umount MGS complete [ 2458.748950] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:35:45 (1786311345) [ 2482.807670] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 2491.889692] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2526.393520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.500899] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2539.548404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.739515] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2539.780269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2539.839497] Lustre: lustre-MDT0001: new disk, initializing [ 2539.898475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2539.921735] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2539.933251] Lustre: 93556:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2539.946617] LustreError: 93556:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2543.873506] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2553.794209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.003980] Lustre: lustre-OST0000: new disk, initializing [ 2554.011112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2554.194251] Lustre: 94322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2554.283515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.683836] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2562.152779] Lustre: 94696:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009821ba6e x1873083107416832/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786311466 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2570.415546] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2576.972972] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.995698] Lustre: MGS: Client cc235015-2c54-4642-b138-b54c81001f5a (at 0@lo) reconnecting [ 2577.043958] LustreError: 93542:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2577.120099] Lustre: 92824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2577.191356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2577.248829] Lustre: lustre-MDT0000: new disk, initializing [ 2577.411344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.469550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2577.492884] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2578.528167] Lustre: 95390:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000bf2c1718 x1873083107423744/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786311483 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2581.411298] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2583.589186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2583.594727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2583.680194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2602.800974] LustreError: 92817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.12@tcp arrived at 1786311491 with bad export cookie 5972848386456226032 [ 2605.024600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2605.031939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.040322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.043300] Lustre: Skipped 1 previous similar message [ 2608.112979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.222852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2613.232017] Lustre: Skipped 3 previous similar messages [ 2619.365339] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2619.538127] Lustre: server umount lustre-OST0000 complete [ 2623.458031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.458147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2623.461359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2623.461367] Lustre: Skipped 2 previous similar messages [ 2623.479274] Lustre: Skipped 1 previous similar message [ 2628.786707] Lustre: server umount lustre-MDT0000 complete [ 2638.476909] Lustre: server umount lustre-MDT0001 complete [ 2641.953928] Lustre: server umount MGS complete [ 2654.131889] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:39:00 (1786311540) [ 2673.461839] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2675.973664] Key type lgssc unregistered [ 2676.250923] LNet: 97899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2676.259583] LNetError: 97899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2676.309721] LNet: Removed LNI 192.168.203.112@tcp [ 2677.087594] Key type .llcrypt unregistered [ 2677.091941] Key type ._llcrypt unregistered [ 2693.925523] Key type ._llcrypt registered [ 2693.929215] Key type .llcrypt registered [ 2694.126263] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2696.041621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2696.058762] alg: No test for adler32 (adler32-zlib) [ 2697.114938] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2697.333366] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2699.089596] Key type lgssc registered [ 2700.372842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2708.262212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2708.271472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.701494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.708599] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2720.241390] LustreError: 99122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.253829] LustreError: 99122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2722.945588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2723.220646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2727.742360] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2734.699682] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2740.362772] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2747.240252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2747.511060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.208781] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2754.671344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2761.177254] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2774.499985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2774.513909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2774.528312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2775.522480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2775.535377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2775.537891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.723604] Lustre: server umount lustre-OST0000 complete [ 2784.736341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2784.740478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.748351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2784.751011] Lustre: Skipped 1 previous similar message [ 2787.178359] Lustre: server umount lustre-MDT0000 complete [ 2790.887201] LustreError: 99123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.918405] LustreError: 99123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2792.457966] LustreError: 99102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311681 with bad export cookie 5596641853395046570 [ 2792.464563] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2792.486077] LustreError: 99102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2792.702582] Lustre: server umount lustre-MDT0001 complete [ 2807.659640] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2810.031471] Key type lgssc unregistered [ 2810.274559] LNet: 102660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2810.278886] LNetError: 102660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2811.301816] LNet: Removed LNI 192.168.203.112@tcp [ 2812.673517] Key type .llcrypt unregistered [ 2812.682272] Key type ._llcrypt unregistered [ 2827.597144] Key type ._llcrypt registered [ 2827.599939] Key type .llcrypt registered [ 2827.767708] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 2829.165672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2829.177633] alg: No test for adler32 (adler32-zlib) [ 2830.536243] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 2830.774901] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 2832.591171] Key type lgssc registered [ 2833.559168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2838.399391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2838.406613] LustreError: 103720:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2838.479041] LustreError: 103723:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2838.484602] LustreError: 103720:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 2838.491152] LustreError: 103720:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2838.497725] LustreError: 103720:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2838.525456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.031732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.917787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.116384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2853.457713] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2858.158366] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing lsmod [ 2861.734976] LustreError: 104928:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2861.742854] LustreError: 104929:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2861.748722] LustreError: 104928:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2861.756914] LustreError: 104928:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2861.762370] LustreError: 104928:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2861.767220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.186661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2867.680675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.686387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2867.702247] Lustre: Skipped 1 previous similar message [ 2867.716502] Lustre: Skipped 1 previous similar message [ 2871.147993] Lustre: server umount lustre-MDT0000 complete [ 2872.810731] LustreError: 103756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.859823] LustreError: 103756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2873.830351] LustreError: 105114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.854375] LustreError: 105114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2875.522100] LustreError: 103740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311764 with bad export cookie 13377354068658020177 [ 2875.533032] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.536039] LustreError: 103740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2875.894282] LustreError: 104191:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000072709b10 x1873083597083136/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2875.996449] Lustre: server umount lustre-MDT0001 complete [ 2891.020314] Lustre: server umount lustre-OST0000 complete [ 2900.560648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.819379] LustreError: 105993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.879657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.087655] LustreError: 105992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.531238] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2911.222214] LustreError: 105993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.298939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.625055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.871674] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2930.206381] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2936.130719] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2942.719858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2943.037738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.176118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2950.713505] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 2957.652391] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2964.449918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2964.458968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.476450] Lustre: Skipped 1 previous similar message [ 2964.485392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2966.498275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2966.502050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.811425] Lustre: server umount lustre-OST0000 complete [ 2974.200198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2974.204836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.214348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.218324] Lustre: Skipped 1 previous similar message [ 2979.091906] Lustre: server umount lustre-MDT0000 complete [ 2981.857383] LustreError: 106979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.878284] LustreError: 106979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2982.346740] LustreError: 105974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786311870 with bad export cookie 13377354068658020891 [ 2982.350934] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2982.358403] LustreError: 105974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2982.627940] Lustre: server umount lustre-MDT0001 complete [ 2993.028762] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 2995.365965] Key type lgssc unregistered [ 2995.576689] LNet: 109528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2995.583660] LNetError: 109528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2996.646541] LNet: Removed LNI 192.168.203.112@tcp [ 2997.426555] Key type .llcrypt unregistered [ 2997.430934] Key type ._llcrypt unregistered [ 3024.252945] Key type ._llcrypt registered [ 3024.255244] Key type .llcrypt registered [ 3024.446413] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3026.339665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3026.354366] alg: No test for adler32 (adler32-zlib) [ 3027.362892] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3027.787165] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 3029.503176] Key type lgssc registered [ 3030.332151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.103111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.117194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.483494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.160233] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3049.956287] LustreError: 110947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.982785] LustreError: 110947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3053.125438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.413967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3057.353842] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3064.749320] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3071.742459] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3080.139935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.436146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.540453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3087.194939] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3094.820547] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3102.792611] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3106.372264] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3110.368926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3110.374825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.389132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3113.442652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3113.443333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.443787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3116.146933] Lustre: server umount lustre-OST0000 complete [ 3120.097407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3120.105075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.117842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3120.122243] Lustre: Skipped 1 previous similar message [ 3123.682475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.687657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.901695] Lustre: server umount lustre-MDT0000 complete [ 3128.805030] LustreError: 112004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.821919] LustreError: 112004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3129.781740] LustreError: 113483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312018 with bad export cookie 13739566850795602795 [ 3129.797622] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.803821] LustreError: 113483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3130.063074] Lustre: server umount lustre-MDT0001 complete [ 3141.688286] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 3143.819657] Key type lgssc unregistered [ 3144.469582] LNet: 114590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3144.490776] LNetError: 114590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3144.518730] LNet: Removed LNI 192.168.203.112@tcp [ 3145.314928] Key type .llcrypt unregistered [ 3145.318263] Key type ._llcrypt unregistered [ 3166.059742] Key type ._llcrypt registered [ 3166.061746] Key type .llcrypt registered [ 3166.401948] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3168.206978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3168.262456] alg: No test for adler32 (adler32-zlib) [ 3169.342561] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3169.559020] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 3171.256858] Key type lgssc registered [ 3172.877943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3181.258397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3181.269527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.618354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.413162] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3192.802363] LustreError: 115815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.822738] LustreError: 115815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3194.213223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.510184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.627984] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3207.046946] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3213.008689] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3220.171696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3220.356524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.115151] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3227.575809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3232.468944] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3245.029341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.043117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3245.045472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3245.072583] Lustre: Skipped 1 previous similar message [ 3249.559464] Lustre: server umount lustre-OST0000 complete [ 3255.266129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.274412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.276390] Lustre: Skipped 1 previous similar message [ 3255.291446] Lustre: Skipped 2 previous similar messages [ 3259.046512] Lustre: server umount lustre-MDT0000 complete [ 3260.388562] LustreError: 115815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.418194] LustreError: 115815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3262.380188] LustreError: 115796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312150 with bad export cookie 5318840737442744045 [ 3262.392381] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3262.396725] LustreError: 115796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3262.655856] Lustre: server umount lustre-MDT0001 complete [ 3273.417931] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 3275.779619] Key type lgssc unregistered [ 3275.979884] LNet: 119349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3275.984992] LNetError: 119349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3276.000866] LNet: Removed LNI 192.168.203.112@tcp [ 3276.866563] Key type .llcrypt unregistered [ 3276.868530] Key type ._llcrypt unregistered [ 3298.892398] Key type ._llcrypt registered [ 3298.894660] Key type .llcrypt registered [ 3298.986626] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3300.559043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.663239] alg: No test for adler32 (adler32-zlib) [ 3301.928740] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3302.161055] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 3303.847933] Key type lgssc registered [ 3304.691911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3311.962356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3311.973314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.217200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3317.136777] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3322.473847] LustreError: 120758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3322.496077] LustreError: 120758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3325.903280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.214354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3330.307959] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3338.257060] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3344.233332] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3352.180460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.566557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.752706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3359.656220] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3367.170300] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3376.708432] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3380.525936] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3380.530596] Lustre: Skipped 2 previous similar messages [ 3384.847962] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3384.853184] Lustre: Skipped 2 previous similar messages [ 3385.955528] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3385.987925] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3395.080723] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.12@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3395.086732] Lustre: Skipped 2 previous similar messages [ 3396.102541] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.12@tcp 192.168.0.[2,10]@tcp [ 3404.101939] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.12/24@tcp in log lustre-MDT0000 [ 3404.110441] Lustre: Skipped 2 previous similar messages [ 3409.926625] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.12/24@tcp [ 3409.932991] Lustre: Skipped 1 previous similar message [ 3417.062684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.071408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.079723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.594389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3421.091257] Lustre: server umount lustre-OST0000 complete [ 3427.810685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.814157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.825522] Lustre: Skipped 2 previous similar messages [ 3431.002200] Lustre: server umount lustre-MDT0000 complete [ 3432.955102] LustreError: 120757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.978911] LustreError: 120757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3435.129786] LustreError: 123388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312323 with bad export cookie 15667286389205459451 [ 3435.133067] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.143412] LustreError: 123388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3435.554665] Lustre: server umount lustre-MDT0001 complete [ 3446.152777] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 3449.041789] Key type lgssc unregistered [ 3449.468425] LNet: 125553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3449.488126] LNetError: 125553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3449.502946] LNet: Removed LNI 192.168.203.112@tcp [ 3450.311499] Key type .llcrypt unregistered [ 3450.313832] Key type ._llcrypt unregistered [ 3486.345210] Key type ._llcrypt registered [ 3486.348223] Key type .llcrypt registered [ 3486.497651] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3488.840112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3488.917259] alg: No test for adler32 (adler32-zlib) [ 3490.006928] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3490.259212] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 3491.903210] Key type lgssc registered [ 3493.248546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3501.115484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3501.125265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.389879] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3502.402620] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.12/24@tcp [ 3502.425897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.369204] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3511.798608] LustreError: 127117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3511.821749] LustreError: 127117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3516.589160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.881936] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3516.950346] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.12/24@tcp [ 3517.055438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3522.526676] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3530.810429] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3537.175774] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3544.394855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.605605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.707411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3550.383771] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3556.811934] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3571.183847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3571.191257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.202336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3572.706488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3572.708116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3572.714711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3572.727639] Lustre: Skipped 1 previous similar message [ 3575.496190] Lustre: server umount lustre-OST0000 complete [ 3582.945601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.947588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3582.960516] Lustre: Skipped 1 previous similar message [ 3582.974030] Lustre: Skipped 1 previous similar message [ 3585.486492] Lustre: server umount lustre-MDT0000 complete [ 3588.066264] LustreError: 127117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.086859] LustreError: 127117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3589.042754] LustreError: 127098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312477 with bad export cookie 12682405001434708900 [ 3589.048228] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.063747] LustreError: 127098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3589.462786] Lustre: server umount lustre-MDT0001 complete [ 3599.842505] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 3602.130242] Key type lgssc unregistered [ 3602.618104] LNet: 130707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3602.631721] LNetError: 130707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3603.704155] LNet: Removed LNI 192.168.203.112@tcp [ 3605.401356] Key type .llcrypt unregistered [ 3605.405716] Key type ._llcrypt unregistered [ 3633.662479] Key type ._llcrypt registered [ 3633.666328] Key type .llcrypt registered [ 3633.785536] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 3648.305672] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3650.137352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3650.221743] alg: No test for adler32 (adler32-zlib) [ 3651.349327] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3651.652947] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 3653.359166] Key type lgssc registered [ 3654.528546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3691.162476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3691.217754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.398078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3692.457733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3692.528830] Lustre: lustre-MDT0000: new disk, initializing [ 3692.584231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.601334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3696.616915] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3708.459783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.593605] Lustre: 134115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3708.713639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3708.719465] Lustre: Skipped 1 previous similar message [ 3708.852261] Lustre: lustre-MDT0001: new disk, initializing [ 3708.959522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.061263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3709.077834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3714.238750] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3722.385440] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3727.652810] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3733.172689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.357737] Lustre: lustre-OST0000: new disk, initializing [ 3733.361270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3733.366714] Lustre: 136082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3733.450935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.680349] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3742.242108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3742.247109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3742.302357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3746.991901] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3751.878398] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3752.051633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3756.330065] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3756.506491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3762.658770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.658899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.677896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3762.698651] Lustre: Skipped 1 previous similar message [ 3762.711376] Lustre: Skipped 1 previous similar message [ 3764.469398] Lustre: server umount lustre-OST0000 complete [ 3767.777793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.780722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.790835] Lustre: Skipped 1 previous similar message [ 3767.802898] Lustre: Skipped 1 previous similar message [ 3772.896406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.900105] Lustre: Skipped 1 previous similar message [ 3773.782952] Lustre: server umount lustre-MDT0000 complete [ 3777.387140] LustreError: 134106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312665 with bad export cookie 13184761502392984810 [ 3777.389509] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3777.398820] LustreError: 134106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3777.782517] Lustre: server umount lustre-MDT0001 complete [ 3785.604258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.912724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.677721] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3791.328717] LustreError: 138301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.338512] LustreError: 138301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3795.496101] LustreError: 138300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3799.004475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.397437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3803.453833] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3810.490495] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3816.359368] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3823.291068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.547642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.885520] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3837.651579] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3842.005351] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3842.218496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3846.198234] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3846.345825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3856.646170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3856.797816] Lustre: lustre-OST0001: new disk, initializing [ 3856.800779] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3856.804755] Lustre: 141674:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3856.867947] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3862.893716] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 3863.263150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3863.274857] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3863.539787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3871.000783] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3871.157511] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3875.688519] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3875.914282] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3880.959566] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a4fc4c57000.ost_server_uuid 50 [ 3882.562814] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a4fc4c57000.ost_server_uuid in FULL state after 0 sec [ 3889.120850] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3889.121268] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.125742] LustreError: Skipped 1 previous similar message [ 3889.128354] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3889.144985] Lustre: Skipped 1 previous similar message [ 3894.243857] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3894.254638] Lustre: Skipped 2 previous similar messages [ 3895.212188] Lustre: server umount lustre-OST0001 complete [ 3899.361143] LustreError: 140125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.366548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.373819] LustreError: 140125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3904.505326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3904.508618] Lustre: Skipped 2 previous similar messages [ 3904.514230] LustreError: 140126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.524546] LustreError: 140126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3905.486085] Lustre: server umount lustre-OST0000 complete [ 3912.160576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3912.170164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.181094] Lustre: Skipped 1 previous similar message [ 3916.243545] Lustre: server umount lustre-MDT0000 complete [ 3919.573778] LustreError: 138282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312808 with bad export cookie 13184761502392985692 [ 3919.582179] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3919.610649] LustreError: 138282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3919.949680] Lustre: server umount lustre-MDT0001 complete [ 3938.468713] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 3940.746427] Key type lgssc unregistered [ 3941.035106] LNet: 144675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3941.041551] LNetError: 144675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3941.052960] LNet: Removed LNI 192.168.203.112@tcp [ 3941.610292] Key type .llcrypt unregistered [ 3941.612099] Key type ._llcrypt unregistered [ 3981.578440] Key type ._llcrypt registered [ 3981.581322] Key type .llcrypt registered [ 3981.737463] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 3994.888152] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 3997.162841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3997.187520] alg: No test for adler32 (adler32-zlib) [ 3998.330960] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 3998.693954] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 4000.439178] Key type lgssc registered [ 4001.425742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4035.086841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4035.099596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.427667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4036.487859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4036.584232] Lustre: lustre-MDT0000: new disk, initializing [ 4036.661879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.677590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4040.794315] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4051.357312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.447403] Lustre: 148824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4051.496412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4051.500174] Lustre: Skipped 1 previous similar message [ 4051.550979] Lustre: lustre-MDT0001: new disk, initializing [ 4051.620160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4051.648075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4051.658587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4056.212822] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4065.431746] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4071.275357] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4077.513608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4077.702280] Lustre: lustre-OST0000: new disk, initializing [ 4077.705583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4077.710850] Lustre: 150795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4077.790817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4084.199546] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4086.827833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4086.885261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4086.962621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4093.732476] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4100.002452] Lustre: Failing over lustre-OST0000 [ 4100.112925] Lustre: server umount lustre-OST0000 complete [ 4117.983237] Lustre: 147048:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312990/real 1786312990] req@000000003082e932 x1873084821831808/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786313006 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4118.033343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4118.727880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4119.075679] Lustre: 147047:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312991/real 1786312991] req@0000000061f29eea x1873084821832832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786313007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4119.112285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.152549] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4119.524873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4119.574332] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4120.159657] Lustre: 147045:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312991/real 1786312991] req@000000002b8148cc x1873084821831936/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786313007 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4123.365438] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4124.383241] Lustre: 147047:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312996/real 1786312996] req@000000007b0a4c55 x1873084821833984/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786313012 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4124.402497] Lustre: 147047:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4126.298459] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4128.607182] Lustre: 147045:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786313001/real 1786313001] req@00000000595defaa x1873084821835136/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786313017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4128.638947] Lustre: 147045:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4128.926538] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4128.926782] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4130.603883] Lustre: Failing over lustre-MDT0000 [ 4130.910939] Lustre: server umount lustre-MDT0000 complete [ 4132.647533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4132.673309] Lustre: Skipped 2 previous similar messages [ 4137.954155] LustreError: 149634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.972731] LustreError: 149634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4139.049637] LustreError: 149943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.082235] LustreError: 148838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.099456] LustreError: 148838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4148.199267] LustreError: 148833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.240684] LustreError: 148833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4149.215902] Lustre: 147045:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786313021/real 1786313021] req@000000001f5d3f52 x1873084821847040/t0(0) o400->MGC192.168.203.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786313037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4149.242907] Lustre: 147045:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4149.249735] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.345322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.336254] LustreError: 149634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.347054] LustreError: 149634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4158.432871] LustreError: 147044:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d74de6c8 x1873084821852544/t0(0) o250->MGC192.168.203.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4158.697764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.771903] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4159.530408] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4162.493205] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4164.086440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4164.091163] Lustre: Skipped 1 previous similar message [ 4164.104501] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4164.182619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4169.698959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4169.704393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.723972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4170.208463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4174.310290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4174.316361] Lustre: Skipped 2 previous similar messages [ 4175.837172] Lustre: server umount lustre-OST0000 complete [ 4184.550664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.564482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4184.573342] Lustre: Skipped 1 previous similar message [ 4184.578943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4184.612245] Lustre: Skipped 1 previous similar message [ 4189.666234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4189.680071] Lustre: Skipped 1 previous similar message [ 4194.271333] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4194.396894] Lustre: server umount lustre-MDT0000 complete [ 4194.798993] LustreError: 148838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.909144] LustreError: 148838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4198.128088] LustreError: 148816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313086 with bad export cookie 9332665597020067669 [ 4198.142050] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.154466] LustreError: 148816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4198.529243] Lustre: server umount lustre-MDT0001 complete [ 4210.311760] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 4213.123570] Key type lgssc unregistered [ 4213.500400] LNet: 154455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4213.512300] LNetError: 154455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4213.527675] LNet: Removed LNI 192.168.203.112@tcp [ 4214.599437] Key type .llcrypt unregistered [ 4214.601509] Key type ._llcrypt unregistered [ 4252.024095] Key type ._llcrypt registered [ 4252.033691] Key type .llcrypt registered [ 4252.250721] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 4253.292821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4253.381494] alg: No test for adler32 (adler32-zlib) [ 4254.423429] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 4254.752730] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 4256.383165] Key type lgssc registered [ 4257.411502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4263.785486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4263.799121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.138800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4269.363906] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4274.658787] LustreError: 155973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.694493] LustreError: 155973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4279.078621] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.777785] LustreError: 155974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.255936] LustreError: 156002:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4282.263959] LustreError: 156002:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4282.270798] LustreError: 156002:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4282.614821] Lustre: server umount lustre-MDT0000 complete [ 4297.444518] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 4305.754230] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 4340.605826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4340.978977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4340.997345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4341.044277] Lustre: lustre-MDT0000: new disk, initializing [ 4341.108079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.131549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4346.577250] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4356.695426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.814692] Lustre: 159306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4356.860770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4356.864885] Lustre: Skipped 1 previous similar message [ 4356.925908] Lustre: lustre-MDT0001: new disk, initializing [ 4357.008380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4357.046727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4357.066310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4361.200838] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4369.252907] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4375.204054] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4381.787796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.049388] Lustre: lustre-OST0000: new disk, initializing [ 4382.052787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4382.057819] Lustre: 161275:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4382.153211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.377889] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4392.095295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4392.105176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4392.340052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4397.973703] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4401.570665] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4403.660276] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:08:10 (1786313290) [ 4638.175778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.176415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.182517] Lustre: Skipped 2 previous similar messages [ 4638.188646] Lustre: Skipped 2 previous similar messages [ 4643.298965] LustreError: 163044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.313663] LustreError: 163044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4643.413293] Lustre: server umount lustre-MDT0000 complete [ 4646.800891] LustreError: 159296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313535 with bad export cookie 11287179367009329385 [ 4646.800891] LustreError: 159298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313535 with bad export cookie 11287179367009329385 [ 4646.806407] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4646.806987] LustreError: 159296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4648.415758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.415803] LustreError: 163063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.422575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4648.441218] LustreError: 163063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4651.423498] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4656.580119] LustreError: 163044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.580255] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4656.599070] LustreError: 163044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4661.215427] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4661.447071] Lustre: server umount lustre-MDT0001 complete [ 4671.628783] Lustre: server umount lustre-OST0000 complete [ 4681.440490] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 4689.782099] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 4724.872920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.035495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4725.060751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4725.111179] Lustre: lustre-MDT0000: new disk, initializing [ 4725.184418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.211351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4729.753319] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4739.286674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4739.472680] Lustre: 165936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4739.524378] LustreError: 165947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.550849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4739.557085] Lustre: Skipped 1 previous similar message [ 4739.612464] Lustre: lustre-MDT0001: new disk, initializing [ 4739.685710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4739.712891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4739.725682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4743.985287] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4752.064664] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4756.624320] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4763.215235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.013569] Lustre: lustre-OST0000: new disk, initializing [ 4764.018614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4764.031301] Lustre: 167903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4764.151501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.811705] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4771.866392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4771.942833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4772.028748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4778.759066] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4783.343170] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4783.496454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4788.219930] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4788.558656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4792.805880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4792.806143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4792.808938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4792.826539] LustreError: Skipped 1 previous similar message [ 4792.845094] Lustre: Skipped 1 previous similar message [ 4803.046158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4803.051656] Lustre: Skipped 4 previous similar messages [ 4805.087301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4805.215681] Lustre: server umount lustre-OST0000 complete [ 4811.233824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4811.247456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.063831] Lustre: server umount lustre-MDT0000 complete [ 4818.416162] LustreError: 165947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.440194] LustreError: 165947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4818.973243] LustreError: 165928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313707 with bad export cookie 11287179367009488215 [ 4818.982941] LustreError: 165928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4818.986960] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4819.204620] Lustre: server umount lustre-MDT0001 complete [ 4828.638815] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:15:15 (1786313715) [ 4845.923196] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 4856.022848] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 4892.074429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4892.246499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4892.269791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4892.323819] Lustre: lustre-MDT0000: new disk, initializing [ 4892.373876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4892.389764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4896.706253] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4906.259226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.539460] Lustre: 172662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4906.795955] LustreError: 172671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.051654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4912.020367] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4921.136444] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4926.230711] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4932.075918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4932.264680] Lustre: lustre-OST0000: new disk, initializing [ 4932.267627] Lustre: Skipped 1 previous similar message [ 4932.270877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4932.275126] Lustre: Skipped 2 previous similar messages [ 4932.278908] Lustre: 174638:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4938.009651] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 4941.911541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4941.923631] Lustre: Skipped 1 previous similar message [ 4941.935957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4942.030747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4948.999547] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4957.666758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4957.680239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.708723] Lustre: Skipped 2 previous similar messages [ 4957.714278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4957.719786] Lustre: Skipped 3 previous similar messages [ 4961.224599] Lustre: server umount lustre-OST0000 complete [ 4967.916630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4967.962629] Lustre: Skipped 1 previous similar message [ 4971.319360] Lustre: server umount lustre-MDT0000 complete [ 4973.031376] LustreError: 173934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.045926] LustreError: 173934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4975.330558] LustreError: 172652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313863 with bad export cookie 11287179367009489097 [ 4975.336183] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4975.341174] LustreError: 172652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4975.640597] Lustre: server umount lustre-MDT0001 complete [ 4984.610561] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:17:51 (1786313871) [ 5000.342738] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 5010.318202] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5045.344563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.500869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5045.518662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5045.587855] Lustre: lustre-MDT0000: new disk, initializing [ 5045.652758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.658202] Lustre: Skipped 2 previous similar messages [ 5045.670577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5050.294681] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5060.091058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.334628] Lustre: 178965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5060.627803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5065.118854] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5074.509837] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5081.020951] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5087.840164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5088.105649] Lustre: 180942:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5093.743435] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5093.981407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5094.302825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5103.474960] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5114.848323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.852142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.864021] LustreError: Skipped 2 previous similar messages [ 5114.898054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5114.968376] Lustre: Skipped 1 previous similar message [ 5114.981505] Lustre: Skipped 4 previous similar messages [ 5124.063127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5124.196437] Lustre: server umount lustre-OST0000 complete [ 5135.025759] Lustre: server umount lustre-MDT0000 complete [ 5135.843081] LustreError: 179847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.878993] LustreError: 179847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5138.928395] LustreError: 178955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314027 with bad export cookie 11287179367009490168 [ 5138.930208] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5138.944716] LustreError: 178955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5139.410558] Lustre: server umount lustre-MDT0001 complete [ 5149.351994] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5151.688324] Key type lgssc unregistered [ 5151.995860] LNet: 182803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5152.004555] LNetError: 182803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5152.016786] LNet: Removed LNI 192.168.203.112@tcp [ 5152.838931] Key type .llcrypt unregistered [ 5152.841093] Key type ._llcrypt unregistered [ 5175.404668] Key type ._llcrypt registered [ 5175.406623] Key type .llcrypt registered [ 5175.525243] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5176.263770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5176.357031] alg: No test for adler32 (adler32-zlib) [ 5177.411525] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5177.633054] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 5179.287132] Key type lgssc registered [ 5180.201866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5187.184815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5187.197983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5188.892725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.936694] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5199.330038] LustreError: 184221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5199.383206] LustreError: 184221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5200.356144] LustreError: 184220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.757112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.966287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5204.784093] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5212.249990] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5217.622959] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5224.247906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5224.466107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.727524] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5234.876869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5238.343701] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5260.258936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.262070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5260.272259] Lustre: Skipped 1 previous similar message [ 5260.293370] Lustre: Skipped 1 previous similar message [ 5262.840844] Lustre: server umount lustre-OST0000 complete [ 5267.427296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5267.437518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.461242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.498462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.498772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.466918] Lustre: server umount lustre-MDT0000 complete [ 5275.616274] LustreError: 184220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.629451] LustreError: 184220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5276.680370] LustreError: 184200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314165 with bad export cookie 12821602322296286741 [ 5276.686816] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.687145] LustreError: 184200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5277.003434] Lustre: server umount lustre-MDT0001 complete [ 5288.494947] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 5290.649044] Key type lgssc unregistered [ 5290.884136] LNet: 187779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5290.892959] LNetError: 187779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5291.944497] LNet: Removed LNI 192.168.203.112@tcp [ 5293.140327] Key type .llcrypt unregistered [ 5293.142875] Key type ._llcrypt unregistered [ 5317.493921] Key type ._llcrypt registered [ 5317.498691] Key type .llcrypt registered [ 5317.587139] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5319.156286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5319.596315] alg: No test for adler32 (adler32-zlib) [ 5320.657508] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 5320.922470] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 5322.575702] Key type lgssc registered [ 5324.138332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5332.103861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5332.124779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.947206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.690885] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5344.236083] LustreError: 189194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.265928] LustreError: 189194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5345.457773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5345.730343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5350.708805] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5357.873984] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5363.729888] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5370.602601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5370.841142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5377.546538] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5379.061452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5383.758576] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5394.402113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5394.417616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.432597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5396.453596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5396.465118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5396.470786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.923793] Lustre: server umount lustre-OST0000 complete [ 5406.596168] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a4fc788a000.ost_server_uuid 50 [ 5408.399924] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5422.050025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5422.058836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5422.063501] Lustre: Skipped 2 previous similar messages [ 5427.169348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5427.177146] Lustre: Skipped 2 previous similar messages [ 5427.545207] Lustre: server umount lustre-MDT0000 complete [ 5431.189176] LustreError: 189175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314319 with bad export cookie 465703692049663124 [ 5431.205028] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.206070] LustreError: 189175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5431.584782] Lustre: server umount lustre-MDT0001 complete [ 5439.213339] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:25:25 (1786314325) [ 5444.845522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.100598] LustreError: 193108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5445.143639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5448.917684] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5450.211385] LustreError: 193109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.333598] LustreError: 193108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.928913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.144383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5461.269366] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5468.535691] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5474.520165] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5481.716489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5481.957498] LustreError: 194929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5482.111841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.496971] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5489.675812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5497.134574] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5504.374136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5504.617782] Lustre: lustre-OST0001: new disk, initializing [ 5504.620906] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5504.625726] Lustre: 195948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5504.694233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5511.051698] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5512.819248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5512.837199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5512.951541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5521.962541] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5528.553505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5528.573378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.605706] Lustre: Skipped 1 previous similar message [ 5528.613715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5528.620433] Lustre: Skipped 1 previous similar message [ 5532.435893] Lustre: server umount lustre-OST0000 complete [ 5533.690159] LustreError: 194929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.714332] LustreError: 194929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5538.338261] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5538.479634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5543.145508] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5543.400161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5554.157342] LustreError: 194929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.196301] LustreError: 194929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5559.266297] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5559.271057] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.277451] Lustre: Skipped 1 previous similar message [ 5559.280487] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5559.286754] Lustre: Skipped 1 previous similar message [ 5562.318604] Lustre: server umount lustre-OST0001 complete [ 5569.508775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.517563] Lustre: Skipped 2 previous similar messages [ 5572.491222] Lustre: server umount lustre-MDT0000 complete [ 5576.500949] LustreError: 193090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314465 with bad export cookie 465703692049664188 [ 5576.535552] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5576.568729] LustreError: 193090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5576.908758] Lustre: server umount lustre-MDT0001 complete [ 5601.896607] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:28:08 (1786314488) [ 5609.826195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.162487] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5610.205752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5610.392616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.359206] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5622.209819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.265391] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5622.289775] Lustre: 199385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5626.745470] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5633.936990] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5640.780919] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5647.275518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5647.385514] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5654.075116] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5654.121516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5662.881591] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5670.417530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5670.473901] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5676.932761] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5678.141187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5684.205196] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5688.294940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.297738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5688.328891] Lustre: Skipped 1 previous similar message [ 5688.345543] Lustre: Skipped 3 previous similar messages [ 5693.452509] Lustre: server umount lustre-OST0000 complete [ 5695.535055] LustreError: 201235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.547270] LustreError: 201235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5708.768331] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.781417] LustreError: Skipped 2 previous similar messages [ 5714.874588] Lustre: server umount lustre-OST0001 complete [ 5724.128276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5724.139932] Lustre: Skipped 10 previous similar messages [ 5724.709584] Lustre: server umount lustre-MDT0000 complete [ 5728.079331] LustreError: 199379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314616 with bad export cookie 465703692049665602 [ 5728.090101] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5728.106278] LustreError: 199379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5728.340519] Lustre: server umount lustre-MDT0001 complete [ 5753.594917] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:30:40 (1786314640) [ 5770.705135] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 5780.124688] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 5812.933488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.146183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5813.181976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5813.229278] Lustre: lustre-MDT0000: new disk, initializing [ 5813.279414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.283505] Lustre: Skipped 3 previous similar messages [ 5813.294691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5817.627389] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5826.727519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.840395] Lustre: 207484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5826.898774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5826.903694] Lustre: Skipped 1 previous similar message [ 5826.969854] Lustre: lustre-MDT0001: new disk, initializing [ 5827.127182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5827.143806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5832.390378] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5841.359659] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5847.398905] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5854.007607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5854.261450] Lustre: lustre-OST0000: new disk, initializing [ 5854.265796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5854.271647] Lustre: 209453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5860.096275] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5862.519706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5862.571105] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5862.925836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5869.611587] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5873.441564] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5873.675466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5877.623678] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5877.816358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5883.365106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5883.380154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5883.381732] LustreError: Skipped 1 previous similar message [ 5883.386232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.422768] Lustre: Skipped 5 previous similar messages [ 5885.485700] Lustre: server umount lustre-OST0000 complete [ 5898.596334] Lustre: server umount lustre-MDT0000 complete [ 5898.739143] LustreError: 207496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5898.769108] LustreError: 207496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5902.534703] LustreError: 210994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314791 with bad export cookie 465703692049667387 [ 5902.596538] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.600760] LustreError: 210994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5902.937572] Lustre: server umount lustre-MDT0001 complete [ 5911.284826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.157852] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5924.437138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.374346] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5936.524887] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5943.502184] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5949.732515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5949.959128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.962661] Lustre: Skipped 4 previous similar messages [ 5955.646644] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 5963.503427] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5967.955942] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5968.398705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5973.136478] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5973.329770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5978.645687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5978.657507] LustreError: Skipped 1 previous similar message [ 5978.665437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.681083] Lustre: Skipped 2 previous similar messages [ 5989.343197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5989.494895] Lustre: server umount lustre-OST0000 complete [ 5995.969582] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6000.991415] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6029.540905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6035.051812] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.081511] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6035.199956] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6040.289947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.348081] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6044.264967] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6047.932833] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6053.917448] Lustre: server umount lustre-OST0000 complete [ 6058.470681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6058.485563] LustreError: Skipped 1 previous similar message [ 6058.498312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6058.509660] Lustre: Skipped 8 previous similar messages [ 6068.242569] Lustre: server umount lustre-MDT0000 complete [ 6071.269236] LustreError: 211667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.297728] LustreError: 211667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6071.681721] LustreError: 214781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314960 with bad export cookie 465703692049668269 [ 6071.685244] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6071.697268] LustreError: 214781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6082.205784] Lustre: server umount lustre-MDT0001 complete [ 6090.924595] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:36:17 (1786314977) [ 6097.515328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.038388] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6109.559919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.049335] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6122.149507] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6127.748414] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6134.885201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6143.181552] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6151.411425] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6155.697696] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6155.845588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6160.072167] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6160.257952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6167.119311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6167.320808] Lustre: lustre-OST0001: new disk, initializing [ 6167.324516] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6167.330039] Lustre: 221010:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6174.071547] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6177.472051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6177.489753] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6177.687147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6182.938224] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6187.537426] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6187.731388] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6191.650775] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6191.936573] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6198.240331] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6198.269959] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.279071] LustreError: Skipped 1 previous similar message [ 6198.313234] Lustre: Skipped 3 previous similar messages [ 6200.482708] Lustre: server umount lustre-OST0001 complete [ 6206.895623] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6207.116241] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6212.090884] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6212.265990] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6241.585407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.742108] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6241.747097] Lustre: Skipped 5 previous similar messages [ 6246.888048] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6246.934903] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6246.936708] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6246.957587] Lustre: Skipped 1 previous similar message [ 6248.387449] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6255.344342] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6257.172466] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6265.098447] Lustre: server umount lustre-OST0001 complete [ 6287.110721] Lustre: server umount lustre-OST0000 complete [ 6297.146445] Lustre: server umount lustre-MDT0000 complete [ 6300.626497] LustreError: 217621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315189 with bad export cookie 465703692049669235 [ 6300.637218] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.638197] LustreError: 217621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6306.945885] Lustre: server umount lustre-MDT0001 complete [ 6331.632930] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:40:17 (1786315217) [ 6337.321234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6337.607035] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6337.657836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6342.774851] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6351.399894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.464298] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6351.480576] Lustre: 226250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6355.733690] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6363.029519] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6368.583704] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6374.746280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6374.872764] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6380.646953] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6388.380529] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6397.743148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6397.786741] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6405.061900] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6414.036383] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6416.066635] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6420.531342] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6420.810611] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6424.847099] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6425.041591] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6429.997599] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a4fc9f9b800.ost_server_uuid 50 [ 6432.001195] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a4fc9f9b800.ost_server_uuid in IDLE state after 0 sec [ 6432.768804] Lustre: Permanently deactivating lustre-OST0001 [ 6432.780792] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6436.178451] Lustre: Permanently reactivating lustre-OST0001 [ 6440.937516] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6440.939264] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6440.943785] LustreError: Skipped 3 previous similar messages [ 6440.973111] Lustre: Skipped 15 previous similar messages [ 6445.772554] Lustre: server umount lustre-OST0001 complete [ 6446.055541] LustreError: 228096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.079367] LustreError: 228096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 6456.824657] Lustre: server umount lustre-OST0000 complete [ 6461.409358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.416030] Lustre: Skipped 12 previous similar messages [ 6466.362945] Lustre: server umount lustre-MDT0000 complete [ 6471.372985] LustreError: 226991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315359 with bad export cookie 465703692049670523 [ 6471.373852] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6471.381875] LustreError: 226991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6471.604600] Lustre: server umount lustre-MDT0001 complete [ 6497.497248] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:43:04 (1786315384) [ 6505.902919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6506.114335] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6506.130486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6506.137114] Lustre: Skipped 5 previous similar messages [ 6510.744805] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6520.154536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.223088] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6520.239652] Lustre: 232941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6524.423696] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6531.638574] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6537.495340] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6544.672058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6544.880376] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6544.894677] Lustre: Permanently deactivating lustre-OST0000 [ 6544.903289] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6550.797970] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6553.662773] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6557.382980] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6563.302488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.352570] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6569.985949] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6572.298938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6578.818800] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6580.782107] Lustre: Permanently reactivating lustre-OST0000 [ 6580.796562] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6580.809147] Lustre: Skipped 2 previous similar messages [ 6586.900967] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6586.911365] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 0000000004da8fc0, cur 1786315475 deadline 1786315472 last 1786315442 [ 6586.960340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6609.375147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6609.443922] Lustre: server umount lustre-OST0001 complete [ 6627.295260] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6627.388987] Lustre: server umount lustre-OST0000 complete [ 6636.872774] Lustre: server umount lustre-MDT0000 complete [ 6639.687318] LustreError: 232935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315528 with bad export cookie 465703692049672175 [ 6639.716866] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6640.302831] Lustre: server umount lustre-MDT0001 complete [ 6649.500904] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 6651.446506] Key type lgssc unregistered [ 6651.668433] LNet: 237828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6651.684324] LNetError: 237828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6651.721839] LNet: Removed LNI 192.168.203.112@tcp [ 6652.267492] Key type .llcrypt unregistered [ 6652.269406] Key type ._llcrypt unregistered [ 6673.060298] Key type ._llcrypt registered [ 6673.068114] Key type .llcrypt registered [ 6673.214522] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 6674.262101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6674.277542] alg: No test for adler32 (adler32-zlib) [ 6675.294551] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 6675.502464] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 6677.167311] Key type lgssc registered [ 6678.246200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6687.449434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6687.471634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6688.770644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6692.416211] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6698.979629] LustreError: 239275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6698.992615] LustreError: 239275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6699.165433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.246604] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6699.272592] Lustre: 239265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6699.281796] Lustre: Permanently deactivating lustre-MDT0001 [ 6699.294981] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6699.516647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6703.368693] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6703.386023] LustreError: 239306:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6704.076403] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6704.107133] LustreError: 239306:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6704.119543] LustreError: 239306:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -108 [ 6704.131115] LustreError: 239306:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6712.100213] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6717.386303] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6723.818038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6724.045294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6725.112180] LustreError: 241118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.131403] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6730.088708] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6730.217890] LustreError: 241118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6734.317182] LustreError: 241118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6734.327089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6734.332875] LustreError: 241118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6736.585512] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6739.435815] LustreError: 241205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6739.470844] LustreError: 241205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6743.058822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6743.238316] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6748.560808] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6748.684886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6748.718349] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6755.672313] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6759.470086] Lustre: Permanently reactivating lustre-MDT0001 [ 6759.487765] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6759.493788] Lustre: Skipped 2 previous similar messages [ 6764.675860] Lustre: lustre-MDT0001: Client 2747842c-85d9-4d1f-8bbf-1358b7989c12 (at 192.168.203.12@tcp) reconnecting [ 6770.818472] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6774.496090] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6780.007612] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6780.227985] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6783.950240] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6784.111434] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6786.897830] Lustre: Permanently deactivating lustre-MDT0001 [ 6786.921924] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6786.941098] Lustre: Skipped 2 previous similar messages [ 6796.371938] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6799.841475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.847255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6799.848183] Lustre: Skipped 3 previous similar messages [ 6799.857517] Lustre: Skipped 3 previous similar messages [ 6804.965588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6804.977783] Lustre: Skipped 3 previous similar messages [ 6805.958694] Lustre: server umount lustre-MDT0000 complete [ 6809.065300] LustreError: 239257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315697 with bad export cookie 12753696233032225447 [ 6809.072452] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6809.075234] LustreError: 239257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6809.342608] Lustre: server umount lustre-MDT0001 complete [ 6818.971810] Lustre: server umount lustre-OST0000 complete [ 6829.109473] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:48:35 (1786315715) [ 6837.767569] Lustre: server umount lustre-OST0001 complete [ 6843.461544] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 6852.852296] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 6887.798296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6888.001695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6888.005958] Lustre: Skipped 2 previous similar messages [ 6888.024080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6888.074712] Lustre: lustre-MDT0000: new disk, initializing [ 6888.122953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6888.135457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6892.284609] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6901.580201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.687755] Lustre: 247531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6901.757422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6901.761298] Lustre: Skipped 1 previous similar message [ 6901.823584] Lustre: lustre-MDT0001: new disk, initializing [ 6901.895414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6901.923554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6901.934546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6906.123406] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6914.180888] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6919.894957] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6926.197305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.406420] Lustre: lustre-OST0000: new disk, initializing [ 6926.410158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6926.418446] Lustre: 249501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6926.484088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6931.736961] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6931.991832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6931.997675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6932.163966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6940.449433] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.537670] LustreError: 247537:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6950.326364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6950.482922] Lustre: lustre-OST0001: new disk, initializing [ 6950.486224] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6950.492645] Lustre: 250615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6952.620199] LustreError: 247537:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6952.639525] LustreError: 247538:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6956.799478] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 6960.892020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6960.947899] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6961.213261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6962.663236] LustreError: 247538:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6966.702724] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6968.111669] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6970.446656] Lustre: lustre-OST0001: Not available for connect from 192.168.203.12@tcp (stopping) [ 6971.364924] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.365325] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.374420] LustreError: Skipped 1 previous similar message [ 6971.402806] Lustre: Skipped 1 previous similar message [ 6975.539049] Lustre: lustre-OST0001: Not available for connect from 192.168.203.12@tcp (stopping) [ 6975.545819] Lustre: Skipped 2 previous similar messages [ 6975.851602] Lustre: server umount lustre-OST0001 complete [ 6976.488831] LustreError: 249484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.522266] LustreError: 249484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6991.843643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.845912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6991.862273] Lustre: Skipped 1 previous similar message [ 6991.874704] Lustre: Skipped 1 previous similar message [ 6996.965973] LustreError: 249486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.996621] LustreError: 249486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6997.987886] Lustre: server umount lustre-OST0000 complete [ 7002.080946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.083617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7002.095856] Lustre: Skipped 1 previous similar message [ 7002.097821] Lustre: Skipped 2 previous similar messages [ 7007.597975] Lustre: server umount lustre-MDT0000 complete [ 7011.529523] LustreError: 248328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315900 with bad export cookie 12753696233032228261 [ 7011.534294] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7011.538176] LustreError: 248328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7017.850320] Lustre: server umount lustre-MDT0001 complete [ 7028.014716] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7030.326472] Key type lgssc unregistered [ 7030.598612] LNet: 252651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7030.629555] LNetError: 252651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7031.719388] LNet: Removed LNI 192.168.203.112@tcp [ 7032.745785] Key type .llcrypt unregistered [ 7032.749872] Key type ._llcrypt unregistered [ 7072.702328] Key type ._llcrypt registered [ 7072.712576] Key type .llcrypt registered [ 7072.890881] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7074.277416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7074.313891] alg: No test for adler32 (adler32-zlib) [ 7075.296805] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7075.617549] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7077.442806] Key type lgssc registered [ 7078.995215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7087.745529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7087.757243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.930767] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7088.945715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7089.060597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7094.923958] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7103.471711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.548062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7103.572256] Lustre: 254861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7103.901223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7108.056981] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7115.310708] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7121.895299] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7129.792619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7129.935722] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7130.120992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.887785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7136.441767] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7144.355359] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7196.566399] Lustre: server umount lustre-OST0000 complete [ 7201.412880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7212.511167] Lustre: 254183:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786316085/real 1786316085] req@000000003df9ce58 x1873088048311424/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786316101 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7212.557071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.601080] Lustre: 254183:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786316086/real 1786316086] req@00000000c2d13708 x1873088048312320/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786316102 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7213.604365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.657305] Lustre: 254183:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7213.851615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7214.153278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.233353] Lustre: 254184:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786316091/real 1786316091] req@000000001ed435f9 x1873088048313216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786316107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7219.251255] Lustre: 254184:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7221.138883] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7223.289576] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7223.323672] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7224.351199] Lustre: 254182:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786316096/real 1786316096] req@0000000014a7e263 x1873088048314368/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786316112 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7224.533123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7224.591685] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7230.384210] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7270.885957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7270.898366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.907135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7270.953836] Lustre: Skipped 1 previous similar message [ 7276.010859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7276.019286] Lustre: Skipped 1 previous similar message [ 7276.990980] Lustre: server umount lustre-OST0000 complete [ 7283.168411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7283.181291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.205703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.217022] Lustre: Skipped 1 previous similar message [ 7287.022321] Lustre: server umount lustre-MDT0000 complete [ 7291.279104] LustreError: 254855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316179 with bad export cookie 17629414656439413177 [ 7291.288209] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7291.289429] LustreError: 254855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7291.951703] Lustre: server umount lustre-MDT0001 complete [ 7305.796851] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7308.792719] Key type lgssc unregistered [ 7309.056705] LNet: 260297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7309.077261] LNetError: 260297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7309.100934] LNet: Removed LNI 192.168.203.112@tcp [ 7310.074059] Key type .llcrypt unregistered [ 7310.080939] Key type ._llcrypt unregistered [ 7337.212810] Key type ._llcrypt registered [ 7337.214363] Key type .llcrypt registered [ 7337.303871] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7339.966586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7340.044320] alg: No test for adler32 (adler32-zlib) [ 7341.215340] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7341.446866] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7343.239249] Key type lgssc registered [ 7344.109818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7351.505326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7351.519253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7352.996896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7358.099501] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7362.610326] LustreError: 261717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.640713] LustreError: 261717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7366.848336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.252720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7372.280242] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7380.720510] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7386.598396] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7393.674409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.906626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.042287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7400.067701] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7408.875881] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7433.203629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.212366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.213050] Lustre: Skipped 1 previous similar message [ 7438.306965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7438.315677] Lustre: Skipped 1 previous similar message [ 7439.109366] Lustre: server umount lustre-OST0000 complete [ 7443.493351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.504545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7443.519980] Lustre: Skipped 2 previous similar messages [ 7448.559326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7448.570961] Lustre: Skipped 1 previous similar message [ 7449.006325] Lustre: server umount lustre-MDT0000 complete [ 7452.636690] LustreError: 265271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316341 with bad export cookie 9153924270995655227 [ 7452.640471] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7452.782766] LustreError: 265271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7453.128352] Lustre: server umount lustre-MDT0001 complete [ 7464.365825] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7467.427512] Key type lgssc unregistered [ 7467.622653] LNet: 266026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7467.630295] LNetError: 266026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7467.644869] LNet: Removed LNI 192.168.203.112@tcp [ 7468.274926] Key type .llcrypt unregistered [ 7468.277342] Key type ._llcrypt unregistered [ 7487.385887] Key type ._llcrypt registered [ 7487.388665] Key type .llcrypt registered [ 7487.476861] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7488.087625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7488.757801] alg: No test for adler32 (adler32-zlib) [ 7490.307456] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7490.625848] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7492.336467] Key type lgssc registered [ 7493.088721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7500.541618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7500.555617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.818867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.542253] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7512.049878] LustreError: 267249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7512.062584] LustreError: 267249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7515.368696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7515.754571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7520.092248] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7528.307388] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7535.178284] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7541.387311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.610340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.812103] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7547.899188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7556.409772] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.474516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7573.487585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.514533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7576.604673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7576.629740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7576.632341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.597904] Lustre: server umount lustre-OST0000 complete [ 7586.803734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.805542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7586.822542] Lustre: Skipped 1 previous similar message [ 7586.836184] Lustre: Skipped 2 previous similar messages [ 7588.941937] Lustre: server umount lustre-MDT0000 complete [ 7591.909984] LustreError: 267977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7591.926893] LustreError: 267977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7593.224349] LustreError: 267228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316481 with bad export cookie 17672988132388212570 [ 7593.233781] LustreError: 267228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7593.238580] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7593.449844] Lustre: server umount lustre-MDT0001 complete [ 7604.976598] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7607.305595] Key type lgssc unregistered [ 7607.942033] LNet: 270941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7607.947245] LNetError: 270941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7607.959935] LNet: Removed LNI 192.168.203.112@tcp [ 7609.158257] Key type .llcrypt unregistered [ 7609.160824] Key type ._llcrypt unregistered [ 7633.867210] Key type ._llcrypt registered [ 7633.868826] Key type .llcrypt registered [ 7633.975469] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7636.093319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7636.238939] alg: No test for adler32 (adler32-zlib) [ 7637.214643] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7637.444986] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7639.079175] Key type lgssc registered [ 7640.395176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7647.267030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7647.282202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.620363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7652.614798] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7658.985600] LustreError: 272359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7659.028953] LustreError: 272359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7662.188818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.438317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7666.933343] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7675.599940] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7681.831497] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7689.081649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7689.396832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7693.500989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7695.592506] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7703.733966] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7727.585648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7727.588422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7727.603351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7727.617466] Lustre: Skipped 1 previous similar message [ 7727.651515] Lustre: Skipped 1 previous similar message [ 7732.708128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.729889] Lustre: Skipped 1 previous similar message [ 7733.685529] Lustre: server umount lustre-OST0000 complete [ 7737.828936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.833107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7737.846475] Lustre: Skipped 1 previous similar message [ 7737.862524] Lustre: Skipped 1 previous similar message [ 7742.945672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.953591] Lustre: Skipped 1 previous similar message [ 7743.920894] Lustre: server umount lustre-MDT0000 complete [ 7748.066354] LustreError: 273394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.107023] LustreError: 273394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7749.037119] LustreError: 272339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316637 with bad export cookie 12443925126495921834 [ 7749.046323] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.116024] LustreError: 272339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7749.380987] Lustre: server umount lustre-MDT0001 complete [ 7762.046523] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7764.973386] Key type lgssc unregistered [ 7765.503162] LNet: 276668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7765.515750] LNetError: 276668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7765.534959] LNet: Removed LNI 192.168.203.112@tcp [ 7767.073790] Key type .llcrypt unregistered [ 7767.076346] Key type ._llcrypt unregistered [ 7785.869870] Key type ._llcrypt registered [ 7785.872474] Key type .llcrypt registered [ 7785.962455] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7787.721904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7787.846960] alg: No test for adler32 (adler32-zlib) [ 7788.883514] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7789.089457] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7790.815302] Key type lgssc registered [ 7791.999534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7800.943853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7800.955144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7802.395150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7806.972398] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7811.560570] LustreError: 277904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7811.579751] LustreError: 277904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7815.500394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.799731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7820.082534] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7828.512890] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7835.419212] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7843.401959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.784398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.883518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7849.538345] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 7857.530048] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7877.626884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7877.630745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.711222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7877.717971] Lustre: Skipped 1 previous similar message [ 7878.862498] Lustre: server umount lustre-OST0000 complete [ 7887.329535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7887.341537] LustreError: Skipped 1 previous similar message [ 7887.351616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7887.368650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7887.385190] Lustre: Skipped 1 previous similar message [ 7889.518168] Lustre: server umount lustre-MDT0000 complete [ 7892.975619] LustreError: 277898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7893.013214] LustreError: 277898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7893.761589] LustreError: 277870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316782 with bad export cookie 17037961686333336868 [ 7893.782838] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.785480] LustreError: 277870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7894.174658] Lustre: server umount lustre-MDT0001 complete [ 7905.606523] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 7908.022495] Key type lgssc unregistered [ 7908.542242] LNet: 281608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7908.575708] LNetError: 281608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7908.596808] LNet: Removed LNI 192.168.203.112@tcp [ 7909.580208] Key type .llcrypt unregistered [ 7909.588523] Key type ._llcrypt unregistered [ 7922.671471] Key type ._llcrypt registered [ 7922.673458] Key type .llcrypt registered [ 7922.977902] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7943.006722] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 7958.492364] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 7960.557155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7960.665768] alg: No test for adler32 (adler32-zlib) [ 7961.839833] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 7962.104099] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 7963.783237] Key type lgssc registered [ 7965.331521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7998.938279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7998.961795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.193653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8000.210976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8000.262432] Lustre: lustre-MDT0000: new disk, initializing [ 8000.333373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.347607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8004.463328] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8014.750190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8014.942392] Lustre: 285162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8015.037861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8015.092523] Lustre: Skipped 1 previous similar message [ 8015.141881] Lustre: lustre-MDT0001: new disk, initializing [ 8015.312924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8015.374739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8015.380074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8020.713250] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8030.722775] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8036.816316] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8044.171197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8044.430192] Lustre: lustre-OST0000: new disk, initializing [ 8044.433903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8044.440054] Lustre: 287130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8044.519412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8051.236227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8051.240539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8051.425388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8051.946204] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8061.408456] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8066.085868] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8066.454657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8070.917130] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8071.137822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8077.280889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8077.283432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.290055] LustreError: Skipped 1 previous similar message [ 8077.305822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.314664] Lustre: Skipped 1 previous similar message [ 8077.347356] Lustre: Skipped 1 previous similar message [ 8079.475878] Lustre: server umount lustre-OST0000 complete [ 8087.011661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8087.018049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.028690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8089.937557] Lustre: server umount lustre-MDT0000 complete [ 8093.963438] LustreError: 288675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316982 with bad export cookie 4491282369096592698 [ 8093.964982] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8093.972478] LustreError: 288675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8094.244631] Lustre: server umount lustre-MDT0001 complete [ 8103.280183] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:09:49 (1786316989) [ 8110.084338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.403385] LustreError: 289545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.414426] LustreError: 289545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8110.454099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8114.753287] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8115.693345] LustreError: 289546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.803806] LustreError: 289545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8122.828438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8123.077282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8126.625828] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8133.891341] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8139.547890] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8146.479925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8146.717204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.636132] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8160.497508] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8171.490452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8171.493186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.506911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.534928] Lustre: Skipped 2 previous similar messages [ 8171.542191] Lustre: Skipped 1 previous similar message [ 8175.390927] Lustre: server umount lustre-OST0000 complete [ 8181.730343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.737430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8181.745282] Lustre: Skipped 1 previous similar message [ 8181.764386] Lustre: Skipped 2 previous similar messages [ 8186.061381] Lustre: server umount lustre-MDT0000 complete [ 8186.871570] LustreError: 290642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8189.748872] LustreError: 292041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317078 with bad export cookie 4491282369096593580 [ 8189.753833] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8189.765855] LustreError: 292041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8190.155987] Lustre: server umount lustre-MDT0001 complete [ 8200.621815] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 8203.529278] Key type lgssc unregistered [ 8203.888669] LNet: 293082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8203.894036] LNetError: 293082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8203.903831] LNet: Removed LNI 192.168.203.112@tcp [ 8204.550179] Key type .llcrypt unregistered [ 8204.552242] Key type ._llcrypt unregistered [ 8231.767074] Key type ._llcrypt registered [ 8231.768750] Key type .llcrypt registered [ 8231.887838] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 8243.837385] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 8245.095351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8245.131134] alg: No test for adler32 (adler32-zlib) [ 8246.173935] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 8246.354531] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 8248.007175] Key type lgssc registered [ 8249.491723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8292.860457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8292.886598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.235210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8294.265258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8294.353348] Lustre: lustre-MDT0000: new disk, initializing [ 8294.420827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.454323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8298.713223] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8308.714041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.944332] Lustre: 296889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8308.992188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8308.997883] Lustre: Skipped 1 previous similar message [ 8309.109534] Lustre: lustre-MDT0001: new disk, initializing [ 8309.223337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8309.261685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8309.273855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8314.139803] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8323.600583] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8329.157032] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8335.483381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8335.715447] Lustre: lustre-OST2710: new disk, initializing [ 8335.718709] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8335.723717] Lustre: 298857:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8335.806611] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8341.160992] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8342.230521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8342.242519] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8342.364199] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8350.053369] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8357.573815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8357.736560] Lustre: lustre-OST03e8: new disk, initializing [ 8357.741257] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8357.749291] Lustre: 299920:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8357.852662] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8363.779201] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8367.156149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8367.177247] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8367.355154] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8372.165176] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8379.739219] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8379.932078] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8384.266124] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8384.439798] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8389.444871] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8389.649399] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8394.602609] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8394.866316] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8401.368327] Lustre: server umount lustre-MDT0000 complete [ 8401.381102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8401.394930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.431307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8405.415499] LustreError: 297682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317294 with bad export cookie 9168222161443038604 [ 8405.424463] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8405.906242] Lustre: server umount lustre-MDT0001 complete [ 8419.503808] Lustre: server umount lustre-OST2710 complete [ 8433.703888] Lustre: server umount lustre-OST03e8 complete [ 8451.168532] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 8460.050316] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 8492.921864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8493.260353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8493.284593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8493.350100] Lustre: lustre-MDT0000: new disk, initializing [ 8493.438295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.502322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8498.196950] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8508.652235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.772478] Lustre: 305168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8508.830471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8508.834621] Lustre: Skipped 1 previous similar message [ 8508.834789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8508.891178] Lustre: lustre-MDT0001: new disk, initializing [ 8508.999389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8509.045048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8509.069552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8513.092103] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8521.101172] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8525.966373] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8532.600095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8532.834514] Lustre: lustre-OST0000: new disk, initializing [ 8532.837854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8532.844310] Lustre: 307134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8532.979688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8537.936203] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8540.202963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8540.219861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8540.331648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8546.453983] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8551.070671] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8551.266849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8555.823199] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8556.041809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8561.124618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8561.133207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.137192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8561.145787] Lustre: Skipped 3 previous similar messages [ 8561.158853] Lustre: Skipped 1 previous similar message [ 8563.863941] Lustre: server umount lustre-OST0000 complete [ 8570.345641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8570.349567] LustreError: Skipped 1 previous similar message [ 8570.352639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.360230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8573.662371] Lustre: server umount lustre-MDT0000 complete [ 8576.490148] LustreError: 308020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8576.522093] LustreError: 308020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 8577.128830] LustreError: 305163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317465 with bad export cookie 9168222161443040277 [ 8577.144917] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8577.149083] LustreError: 305163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8577.454716] Lustre: server umount lustre-MDT0001 complete [ 8586.304175] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:17:52 (1786317472) [ 8587.574473] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8589.316321] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:17:55 (1786317475) [ 8592.457422] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 8618.591955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8618.759958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8618.774512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8618.881261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8623.196298] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8631.807268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.864564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8631.882642] Lustre: 310990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8636.832597] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8658.557145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8658.676963] Lustre: Denying initial registration attempt from nid 192.168.203.112@tcp, specified as failover [ 8658.685703] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8658.694452] LustreError: 312445:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8658.699687] LustreError: 312445:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8658.703408] LustreError: 312445:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8658.774923] Lustre: server umount lustre-OST0000 complete [ 8658.777863] LustreError: 312445:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8666.086544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.093062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.267070] Lustre: Skipped 2 previous similar messages [ 8666.270185] Lustre: Skipped 2 previous similar messages [ 8671.204602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.226665] Lustre: Skipped 1 previous similar message [ 8671.911599] Lustre: server umount lustre-MDT0000 complete [ 8675.453898] LustreError: 310985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317564 with bad export cookie 9168222161443041166 [ 8675.463220] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8675.463743] LustreError: 310985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8675.751261] Lustre: server umount lustre-MDT0001 complete [ 8687.050087] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 8689.566240] Key type lgssc unregistered [ 8689.925786] LNet: 313570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8689.930625] LNetError: 313570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8689.944749] LNet: Removed LNI 192.168.203.112@tcp [ 8690.850327] Key type .llcrypt unregistered [ 8690.854737] Key type ._llcrypt unregistered [ 8703.557538] Key type ._llcrypt registered [ 8703.559993] Key type .llcrypt registered [ 8703.635298] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 8704.418865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8704.581510] alg: No test for adler32 (adler32-zlib) [ 8705.792959] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 8706.055136] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 8707.680078] Key type lgssc registered [ 8708.654124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8754.953053] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 8764.708052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8764.732577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.917776] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8765.939616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8766.047297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8769.941867] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8778.731237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.784131] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8778.804516] Lustre: 316337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8778.985538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8782.854556] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8790.681403] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8796.210561] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8816.166142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8816.321446] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8816.478396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8822.612207] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8830.007643] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8836.590955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.610161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8836.610620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8836.614354] LustreError: Skipped 1 previous similar message [ 8841.713846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8841.722219] Lustre: Skipped 1 previous similar message [ 8846.857689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8846.861533] Lustre: Skipped 1 previous similar message [ 8847.327235] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8847.452373] Lustre: server umount lustre-OST0000 complete [ 8851.939295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.942501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8851.952188] Lustre: Skipped 2 previous similar messages [ 8851.962481] Lustre: Skipped 2 previous similar messages [ 8857.059491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8857.067560] Lustre: Skipped 1 previous similar message [ 8857.917392] Lustre: server umount lustre-MDT0000 complete [ 8861.254696] LustreError: 317078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317749 with bad export cookie 17920181844325277835 [ 8861.255828] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8861.265831] LustreError: 317078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8861.602903] Lustre: server umount lustre-MDT0001 complete [ 8872.710301] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 8875.208945] Key type lgssc unregistered [ 8875.538780] LNet: 319912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8875.552392] LNetError: 319912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8875.627597] LNet: Removed LNI 192.168.203.112@tcp [ 8876.559827] Key type .llcrypt unregistered [ 8876.562297] Key type ._llcrypt unregistered [ 8903.129848] Key type ._llcrypt registered [ 8903.131803] Key type .llcrypt registered [ 8903.215185] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 8904.402683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8904.441194] alg: No test for adler32 (adler32-zlib) [ 8905.437447] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 8905.728567] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 8907.431295] Key type lgssc registered [ 8908.353521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8916.220331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8916.241399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.958318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8921.807523] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8928.233250] LustreError: 321328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8928.270933] LustreError: 321328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8929.250412] LustreError: 321329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8929.532881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.821666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8934.171851] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8942.589341] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8948.029809] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8956.523244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8956.730316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.193373] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 8969.002819] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8986.080307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8986.088712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8986.097511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.595176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8986.600207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.605754] Lustre: Skipped 1 previous similar message [ 8986.623221] Lustre: Skipped 1 previous similar message [ 8991.714438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8991.757983] Lustre: Skipped 3 previous similar messages [ 8996.836609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8996.841564] Lustre: Skipped 3 previous similar messages [ 8998.367119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8998.535066] Lustre: server umount lustre-MDT0000 complete [ 9001.883188] LustreError: 321310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317890 with bad export cookie 8257197160392317538 [ 9001.892892] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9001.955686] LustreError: 321323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9001.957198] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.978837] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9001.993710] Lustre: Skipped 1 previous similar message [ 9007.072265] LustreError: 321323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9007.080771] LustreError: 321323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9012.193124] LustreError: 321325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9012.193731] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9012.201113] LustreError: 321325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9012.205467] Lustre: Skipped 3 previous similar messages [ 9016.288183] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9016.464767] Lustre: server umount lustre-MDT0001 complete [ 9022.117176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9030.132874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.007901] LustreError: 320638:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000043a3006 x1873089967204480/t0(0) o250->MGC192.168.203.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9047.322209] LustreError: 323161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9047.337394] LustreError: 323161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9047.389895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9047.450223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9051.965542] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9060.850101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.234490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9065.690958] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9067.503388] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9072.548994] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9074.039586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9079.377968] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9081.536934] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9086.950779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9086.959455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.971516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9086.979813] Lustre: Skipped 1 previous similar message [ 9100.255222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9100.375231] Lustre: server umount lustre-OST0000 complete [ 9107.426816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9107.434547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.457905] Lustre: Skipped 1 previous similar message [ 9109.927623] Lustre: server umount lustre-MDT0000 complete [ 9113.489785] LustreError: 321309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318002 with bad export cookie 8257197160392325140 [ 9113.497343] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9113.504438] LustreError: 321309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9113.941903] Lustre: server umount lustre-MDT0001 complete [ 9124.570295] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 9126.635816] Key type lgssc unregistered [ 9126.846795] LNet: 327630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9126.851310] LNetError: 327630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9126.861868] LNet: Removed LNI 192.168.203.112@tcp [ 9127.564298] Key type .llcrypt unregistered [ 9127.566370] Key type ._llcrypt unregistered [ 9152.597378] Key type ._llcrypt registered [ 9152.599612] Key type .llcrypt registered [ 9152.705431] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 9154.412807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9154.842879] alg: No test for adler32 (adler32-zlib) [ 9156.339433] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 9156.860174] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 9158.631156] Key type lgssc registered [ 9159.646568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9167.403575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9167.439291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.796272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9173.223505] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9179.108065] LustreError: 329027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.122814] LustreError: 329027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9180.140174] LustreError: 329026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9180.535215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9180.762837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9185.135740] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9192.822487] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9198.888729] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9205.218143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9205.244854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9205.255130] Lustre: Skipped 1 previous similar message [ 9206.307828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9206.317365] Lustre: Skipped 1 previous similar message [ 9208.596860] Lustre: server umount lustre-MDT0000 complete [ 9211.365578] LustreError: 330034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9211.380581] LustreError: 330034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9211.665190] LustreError: 329008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318100 with bad export cookie 14047744628872619014 [ 9211.667749] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9211.673587] LustreError: 329008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9212.009703] Lustre: server umount lustre-MDT0001 complete [ 9224.679249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.842217] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9224.867278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9225.008727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9228.627083] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9235.642278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.702634] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9235.723839] Lustre: 331613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9235.964588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9240.398146] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9247.135988] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9252.740441] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9262.245034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9262.357729] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9262.367959] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9262.375589] LustreError: 333496:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9262.381905] LustreError: 333496:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9262.386161] LustreError: 333496:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9262.428291] Lustre: server umount lustre-OST0000 complete [ 9262.431862] LustreError: 333496:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9273.137045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9273.283795] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9273.410540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9279.372575] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9282.051521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9287.638843] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9294.708864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9294.799517] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9294.861991] Lustre: lustre-OST0001: new disk, initializing [ 9294.865923] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9294.871153] Lustre: 335041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9294.976470] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9300.765792] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9301.215843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9301.229716] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9301.378087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9309.001660] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9316.840962] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9316.842191] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9316.906374] Lustre: Skipped 1 previous similar message [ 9316.912784] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9316.924433] Lustre: Skipped 1 previous similar message [ 9321.970749] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9326.049690] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9326.098691] Lustre: server umount lustre-OST0001 complete [ 9327.078264] LustreError: 333960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.102894] LustreError: 333960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9332.199192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.199476] LustreError: 334452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9332.202612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9332.202624] Lustre: Skipped 1 previous similar message [ 9332.220843] Lustre: Skipped 2 previous similar messages [ 9332.263597] LustreError: 334452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9335.558181] Lustre: server umount lustre-OST0000 complete [ 9342.433239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.434606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9342.441189] Lustre: Skipped 1 previous similar message [ 9342.447642] Lustre: Skipped 2 previous similar messages [ 9344.920786] Lustre: server umount lustre-MDT0000 complete [ 9347.557145] LustreError: 331625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9347.577404] LustreError: 331625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9348.133908] LustreError: 331607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318236 with bad export cookie 14047744628872619574 [ 9348.136255] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9348.143644] LustreError: 331607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9348.410569] Lustre: server umount lustre-MDT0001 complete [ 9358.489675] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 9360.903882] Key type lgssc unregistered [ 9361.131564] LNet: 337003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9361.141705] LNetError: 337003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9362.226862] LNet: Removed LNI 192.168.203.112@tcp [ 9362.852289] Key type .llcrypt unregistered [ 9362.858132] Key type ._llcrypt unregistered [ 9432.279563] Key type ._llcrypt registered [ 9432.282017] Key type .llcrypt registered [ 9432.376166] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [ 9446.078908] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 9447.305270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9447.336059] alg: No test for adler32 (adler32-zlib) [ 9448.480514] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 9448.669957] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 9450.319186] Key type lgssc registered [ 9451.472297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9481.963195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9482.001702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.251512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9483.281243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9483.391915] Lustre: lustre-MDT0000: new disk, initializing [ 9483.457354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.470775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9487.812237] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9498.503311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.572963] Lustre: 341931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9498.632265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9498.635872] Lustre: Skipped 1 previous similar message [ 9498.706789] Lustre: lustre-MDT0001: new disk, initializing [ 9498.786088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9498.818753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9498.836815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9503.159258] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9512.543039] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9517.856419] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9524.937381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9525.148149] Lustre: lustre-OST0000: new disk, initializing [ 9525.151639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9525.157412] Lustre: 343897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9525.275412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9531.098897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9531.121210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9531.313605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9531.366899] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9540.493221] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9545.771429] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9546.097253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9550.294359] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9550.567365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9556.967318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9556.977338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9556.980711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9557.002041] Lustre: Skipped 1 previous similar message [ 9558.844658] Lustre: server umount lustre-OST0000 complete [ 9565.700099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9565.709903] LustreError: Skipped 1 previous similar message [ 9565.717985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9565.765407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9565.771383] Lustre: Skipped 1 previous similar message [ 9567.244408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.262348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9567.283578] Lustre: Skipped 1 previous similar message [ 9569.195789] Lustre: server umount lustre-MDT0000 complete [ 9572.456941] LustreError: 341922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318461 with bad export cookie 10292713562343932452 [ 9572.466929] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9572.474585] LustreError: 341922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9572.740790] Lustre: server umount lustre-MDT0001 complete [ 9581.399205] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:34:27 (1786318467) [ 9589.559122] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:34:35 (1786318475) [ 9598.124241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.436432] LustreError: 346654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9598.452588] LustreError: 346654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9598.502230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9602.957965] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9603.553203] LustreError: 346655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9608.689517] LustreError: 346654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9610.681687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9610.985675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9614.911207] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9623.983879] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9630.522795] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9637.371738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9637.652362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.116714] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9650.438706] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9662.649573] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9664.682867] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9666.680625] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9668.315493] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9670.604733] Lustre: Failing over lustre-MDT0000 [ 9670.867940] Lustre: server umount lustre-MDT0000 complete [ 9671.649328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9671.661200] LustreError: 346655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9671.695373] Lustre: Skipped 2 previous similar messages [ 9671.735566] LustreError: 346655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9676.773367] LustreError: 346654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9676.800430] LustreError: 346654:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9687.034081] LustreError: 349565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.091414] LustreError: 349565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9687.839278] Lustre: 340153:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786318560/real 1786318560] req@00000000e6ccdc10 x1873090536610176/t0(0) o400->MGC192.168.203.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786318576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9687.881489] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9689.229951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.498042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.534679] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9699.412777] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9702.201558] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9703.942910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9703.986023] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9704.039389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9713.302739] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9715.411349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9718.474318] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9721.042830] Lustre: Failing over lustre-MDT0000 [ 9721.448150] Lustre: server umount lustre-MDT0000 complete [ 9724.389271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9724.392649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9724.396101] LustreError: 349565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9724.396116] LustreError: 349565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 9724.425186] Lustre: Skipped 1 previous similar message [ 9739.743947] Lustre: 340156:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786318612/real 1786318612] req@0000000075555777 x1873090536630784/t0(0) o400->MGC192.168.203.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786318628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9739.790199] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9740.900462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.208722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9750.268974] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9750.618984] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9754.330655] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9755.639646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9755.652950] Lustre: Skipped 2 previous similar messages [ 9755.675706] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9755.735380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9764.498708] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9766.269309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9776.099996] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9776.100784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9776.105756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9776.109825] Lustre: Skipped 2 previous similar messages [ 9778.936473] Lustre: server umount lustre-MDT0000 complete [ 9781.230552] LustreError: 347382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9781.286917] LustreError: 347382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 9783.243748] LustreError: 346634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318671 with bad export cookie 10292713562343935161 [ 9783.252205] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9783.266451] LustreError: 346634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9783.688973] Lustre: server umount lustre-MDT0001 complete [ 9798.248203] Lustre: server umount lustre-OST0000 complete [ 9813.999200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9814.345380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9819.526506] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9827.159391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.487700] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9838.110228] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9843.630000] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9850.548277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9857.097510] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9859.087570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9864.644425] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9873.379380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9873.384443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9873.407798] Lustre: Skipped 1 previous similar message [ 9873.420218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9873.433860] Lustre: Skipped 2 previous similar messages [ 9876.091265] Lustre: server umount lustre-OST0000 complete [ 9884.128690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9884.138100] LustreError: Skipped 1 previous similar message [ 9884.148085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9884.164766] Lustre: Skipped 2 previous similar messages [ 9894.369910] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9894.528959] Lustre: server umount lustre-MDT0000 complete [ 9894.890036] LustreError: 353012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9894.911785] LustreError: 353012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9898.375532] LustreError: 352996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318786 with bad export cookie 10292713562343935910 [ 9898.391543] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9898.394135] LustreError: 352996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9898.654084] Lustre: server umount lustre-MDT0001 complete [ 9911.020589] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [ 9913.439375] Key type lgssc unregistered [ 9913.659803] LNet: 356553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9913.669145] LNetError: 356553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9913.681073] LNet: Removed LNI 192.168.203.112@tcp [ 9915.135831] Key type .llcrypt unregistered [ 9915.144686] Key type ._llcrypt unregistered [ 9942.187491] Key type ._llcrypt registered [ 9942.189670] Key type .llcrypt registered [ 9942.279383] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [ 9944.206389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9944.353084] alg: No test for adler32 (adler32-zlib) [ 9945.435354] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [ 9945.682306] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [ 9947.311687] Key type lgssc registered [ 9948.396600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9956.351751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9956.366191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.666598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9962.045367] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9968.140703] LustreError: 358071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9968.168818] LustreError: 358071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9970.013444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.358517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9974.725552] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [ 9981.722315] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9987.166443] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9995.099082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9995.243523] LustreError: 359891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9995.409137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10001.778971] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10002.436712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10009.593537] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10021.344433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10021.348060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10021.372115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10022.881880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10022.895513] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.655603] Lustre: server umount lustre-MDT0000 complete [10028.004031] LustreError: 358067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10028.027327] LustreError: 358067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10029.685747] LustreError: 358050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318918 with bad export cookie 3678353481694284918 [10029.695232] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10029.702702] LustreError: 358050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10030.182297] Lustre: server umount lustre-MDT0001 complete [10044.532538] Lustre: server umount lustre-OST0000 complete [10057.862942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10058.243702] LustreError: 361783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10058.368497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10062.466190] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10063.841394] LustreError: 361784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10070.005507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10070.239875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10074.319840] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10082.574340] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10088.791577] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10095.012633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10095.086922] LustreError: 363602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.098405] LustreError: 363602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10095.425321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10101.629677] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10102.811341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10108.911462] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10121.697363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10121.703488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.712341] Lustre: Skipped 1 previous similar message [10121.716898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10121.723198] Lustre: Skipped 2 previous similar messages [10125.267136] Lustre: server umount lustre-MDT0000 complete [10128.365118] LustreError: 361783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10128.378904] LustreError: 361783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10129.098127] LustreError: 361764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319017 with bad export cookie 3678353481694286031 [10129.126920] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10129.451683] Lustre: server umount lustre-MDT0001 complete [10144.117273] Lustre: server umount lustre-OST0000 complete [10160.478328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.839225] LustreError: 365693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.903385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10165.629237] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10175.418227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10181.624215] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10188.782406] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10195.291385] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10203.607857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10203.939854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10203.946147] Lustre: Skipped 1 previous similar message [10206.030217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10210.619354] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10218.386902] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10226.658746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10226.660899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10226.666707] Lustre: Skipped 2 previous similar messages [10226.670102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10226.671814] LustreError: Skipped 1 previous similar message [10226.692172] Lustre: Skipped 4 previous similar messages [10228.726854] Lustre: server umount lustre-OST0000 complete [10236.897816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10236.901032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10236.904788] Lustre: Skipped 2 previous similar messages [10247.142266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10247.144886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10247.171500] Lustre: Skipped 4 previous similar messages [10247.442308] Lustre: server umount lustre-MDT0000 complete [10251.025269] LustreError: 365675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319139 with bad export cookie 3678353481694287053 [10251.030868] LustreError: 365675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10251.039143] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10251.296926] Lustre: server umount lustre-MDT0001 complete [10264.410623] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [10267.051878] Key type lgssc unregistered [10267.421748] LNet: 369232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10267.432316] LNetError: 369232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10267.457799] LNet: Removed LNI 192.168.203.112@tcp [10268.162281] Key type .llcrypt unregistered [10268.164715] Key type ._llcrypt unregistered [10295.858874] Key type ._llcrypt registered [10295.861601] Key type .llcrypt registered [10295.961713] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [10296.856743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10297.242846] alg: No test for adler32 (adler32-zlib) [10299.225807] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [10299.684354] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [10301.375188] Key type lgssc registered [10302.185736] Lustre: Echo OBD driver; http://www.lustre.org/ [10310.067863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10310.116138] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.121606] LustreError: 370672:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10310.135601] LustreError: 370672:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10310.152709] LustreError: 370672:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10310.163978] LustreError: 370672:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10310.168775] LustreError: 370672:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10318.523564] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10318.528822] LustreError: 371070:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10318.537988] LustreError: 371070:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10318.543280] LustreError: 371070:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10318.548027] LustreError: 371070:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10318.552900] LustreError: 371070:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10338.786836] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [10341.667267] Key type lgssc unregistered [10341.974994] LNet: 372078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10341.985499] LNetError: 372078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10342.001606] LNet: Removed LNI 192.168.203.112@tcp [10342.759731] Key type .llcrypt unregistered [10342.761877] Key type ._llcrypt unregistered [10362.830626] Key type ._llcrypt registered [10362.834626] Key type .llcrypt registered [10362.915669] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [10376.638551] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [10378.213567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10378.468289] alg: No test for adler32 (adler32-zlib) [10379.807825] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [10380.285784] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [10381.967393] Key type lgssc registered [10383.178241] Lustre: Echo OBD driver; http://www.lustre.org/ [10416.384304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10416.413972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10417.589652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10417.609429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10417.672337] Lustre: lustre-MDT0000: new disk, initializing [10417.749217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10417.849081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10422.568615] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10435.000034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.097177] Lustre: 375216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10435.135562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10435.139104] Lustre: Skipped 1 previous similar message [10435.189329] Lustre: lustre-MDT0001: new disk, initializing [10435.258378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10435.289430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10435.295635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10439.995649] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10450.974841] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10456.935185] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10464.061804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.323805] Lustre: lustre-OST0000: new disk, initializing [10464.331720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10464.337778] Lustre: 377188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10464.422134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10471.819514] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10474.702954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10474.752299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10474.920680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10481.724299] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10486.103924] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10486.358517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10490.466751] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10490.649177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10495.457385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10495.458900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10495.473339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10495.476865] Lustre: Skipped 1 previous similar message [10495.498792] Lustre: Skipped 1 previous similar message [10499.568945] Lustre: server umount lustre-OST0000 complete [10505.697742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10505.700917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10505.704185] Lustre: Skipped 1 previous similar message [10510.489730] Lustre: server umount lustre-MDT0000 complete [10514.200485] LustreError: 375208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319402 with bad export cookie 10526255938637892740 [10514.200485] LustreError: 378728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319402 with bad export cookie 10526255938637892740 [10514.202383] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10514.206739] LustreError: 375208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10514.556112] Lustre: server umount lustre-MDT0001 complete [10522.632170] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10524.302570] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:50:10 (1786319410) [10529.963857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.240391] LustreError: 379647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.251726] LustreError: 379647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10530.299632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10535.396201] LustreError: 379648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.410270] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10539.506106] LustreError: 379647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10543.587547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10543.884679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10548.549202] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10555.819257] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10561.861711] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10569.143086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10569.384676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10576.057894] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10583.701337] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10591.653878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10591.867885] Lustre: lustre-OST0001: new disk, initializing [10591.870768] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10591.875183] Lustre: 382483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10591.951166] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10597.654434] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10600.012029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10600.018580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10600.117024] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10606.176651] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10610.661720] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.662290] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10610.662298] LustreError: Skipped 1 previous similar message [10610.662771] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10610.662775] Lustre: Skipped 1 previous similar message [10610.711769] Lustre: Skipped 1 previous similar message [10613.308220] Lustre: lustre-OST0001: Not available for connect from 192.168.203.12@tcp (stopping) [10613.313384] Lustre: Skipped 1 previous similar message [10616.106606] Lustre: server umount lustre-OST0001 complete [10618.408783] LustreError: 382949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10626.027418] LustreError: 381467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10626.053103] LustreError: 381467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10632.161943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10632.168460] LustreError: Skipped 1 previous similar message [10632.180135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10632.194071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10632.197631] Lustre: Skipped 2 previous similar messages [10635.235183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10636.261656] LustreError: 381469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10636.276218] LustreError: 381469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10638.262388] Lustre: server umount lustre-OST0000 complete [10646.496762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10646.498358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10646.509194] Lustre: Skipped 1 previous similar message [10646.521543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10646.521550] Lustre: Skipped 3 previous similar messages [10647.747670] Lustre: server umount lustre-MDT0000 complete [10651.346746] LustreError: 379628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319539 with bad export cookie 10526255938637893622 [10651.358664] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10657.678404] Lustre: server umount lustre-MDT0001 complete [10667.618261] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [10669.503425] Key type lgssc unregistered [10669.705877] LNet: 384476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10669.714955] LNetError: 384476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10669.732303] LNet: Removed LNI 192.168.203.112@tcp [10670.464869] Key type .llcrypt unregistered [10670.467334] Key type ._llcrypt unregistered [10721.752455] Key type ._llcrypt registered [10721.754561] Key type .llcrypt registered [10721.843400] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [10722.926259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10723.145171] alg: No test for adler32 (adler32-zlib) [10724.747857] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [10725.487569] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [10727.223199] Key type lgssc registered [10728.477489] Lustre: Echo OBD driver; http://www.lustre.org/ [10737.071651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10737.130207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.350376] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10738.368623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10738.506542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10744.750875] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10754.382656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10754.437752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10754.466516] Lustre: 387276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10754.879882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10760.124554] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10767.236806] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10772.799535] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10775.524673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10775.529760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.539787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10778.593068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10778.595516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10780.686338] Lustre: server umount lustre-MDT0000 complete [10784.090907] LustreError: 387266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319672 with bad export cookie 3985109395228190708 [10784.092782] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10784.099105] LustreError: 387266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10784.318623] Lustre: server umount lustre-MDT0001 complete [10796.378327] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:54:42 (1786319682) [10802.258178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10802.685542] LustreError: 389788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.697074] LustreError: 389788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10802.736940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10807.356850] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10807.778411] LustreError: 389789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10812.897710] LustreError: 389788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10815.730339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10815.945360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10820.156629] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10826.805563] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10832.621837] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10839.047812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10839.160349] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10839.298490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10845.110679] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10851.861273] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10860.105661] Lustre: Permanently deactivating lustre-OST0000 [10860.128782] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10869.827604] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10872.873755] LustreError: 392871:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10872.879090] LustreError: 392871:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10872.882838] LustreError: 392871:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10875.625340] LustreError: 392919:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10875.689356] LustreError: 392919:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10883.712922] Lustre: server umount lustre-OST0000 complete [10887.448552] LustreError: 393169:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10887.457394] LustreError: 393169:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10889.185360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.191429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10889.216782] Lustre: Skipped 1 previous similar message [10889.239565] Lustre: Skipped 2 previous similar messages [10894.304971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10894.312980] Lustre: Skipped 1 previous similar message [10899.425615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10899.430248] Lustre: Skipped 1 previous similar message [10899.493302] Lustre: server umount lustre-MDT0000 complete [10902.749522] LustreError: 389769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319791 with bad export cookie 3985109395228191289 [10902.756191] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10902.777528] LustreError: 389769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10913.380346] Lustre: server umount lustre-MDT0001 complete [10922.838960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.287344] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10931.150172] LustreError: 394332:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10936.647386] Lustre: 394671:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.112@tcp [10938.618622] Lustre: server umount MGS complete [10950.000611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10950.576689] LustreError: 395393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.599730] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10950.604124] Lustre: Skipped 1 previous similar message [10950.642048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10955.242688] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10955.746473] LustreError: 395394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.275973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10962.451526] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10967.559990] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [10976.422637] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10981.836816] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10983.665514] Lustre: Permanently reactivating lustre-OST0000 [10983.684209] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10983.689574] Lustre: Skipped 2 previous similar messages [11013.599181] LustreError: 397240:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11021.854453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.103733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11022.108368] Lustre: Skipped 1 previous similar message [11028.119374] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11035.634624] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11042.693551] Lustre: server umount lustre-OST0000 complete [11049.441327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11049.452290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11049.475436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11056.188351] Lustre: server umount lustre-MDT0000 complete [11060.120361] LustreError: 396134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319948 with bad export cookie 3985109395228192339 [11060.139087] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11060.152294] LustreError: 396134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11069.556210] Lustre: server umount lustre-MDT0001 complete [11081.389924] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [11083.592695] Key type lgssc unregistered [11083.792282] LNet: 399903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11083.798802] LNetError: 399903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11083.812171] LNet: Removed LNI 192.168.203.112@tcp [11084.818383] Key type .llcrypt unregistered [11084.820397] Key type ._llcrypt unregistered [11103.396310] Key type ._llcrypt registered [11103.399606] Key type .llcrypt registered [11103.507781] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [11118.343595] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [11119.549689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11119.854429] alg: No test for adler32 (adler32-zlib) [11121.070399] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [11121.285166] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [11123.095176] Key type lgssc registered [11124.184980] Lustre: Echo OBD driver; http://www.lustre.org/ [11157.539384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11157.566712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.735146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11158.757848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11158.833376] Lustre: lustre-MDT0000: new disk, initializing [11158.905232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11158.926766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11163.395495] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11176.313138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11176.422323] Lustre: 403060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11176.470703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11176.475500] Lustre: Skipped 1 previous similar message [11176.576619] Lustre: lustre-MDT0001: new disk, initializing [11176.697379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11176.735586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11176.754817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11181.280588] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11190.915602] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11196.718449] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11204.106381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11204.390118] Lustre: lustre-OST0000: new disk, initializing [11204.412919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11204.426584] Lustre: 405040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11204.587338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11210.683799] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11211.823365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11211.830920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11212.006154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11222.140913] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11227.056985] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11227.230903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11231.780682] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11232.010524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11237.863367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11237.870720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.890435] Lustre: Skipped 1 previous similar message [11237.902356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11237.937367] Lustre: Skipped 1 previous similar message [11240.264164] Lustre: server umount lustre-OST0000 complete [11246.052107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11246.116424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11246.120768] Lustre: Skipped 1 previous similar message [11247.169164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11247.173413] Lustre: Skipped 2 previous similar messages [11251.848658] Lustre: server umount lustre-MDT0000 complete [11255.762077] LustreError: 403874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320144 with bad export cookie 2572784338787894444 [11255.771115] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11255.773989] LustreError: 403874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11256.010222] Lustre: server umount lustre-MDT0001 complete [11266.391781] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11267.823804] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:02:34 (1786320154) [11275.268940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.715891] LustreError: 407503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11275.738618] LustreError: 407503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11275.809524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11280.141410] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11280.866642] LustreError: 407504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11285.991865] LustreError: 407503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.015098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.299209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11292.861644] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11300.658818] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11306.699739] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11313.100083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11313.337426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11319.516078] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11327.116721] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11328.973236] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11331.967226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11343.329469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11343.333686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11343.334894] LustreError: Skipped 1 previous similar message [11343.335845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11343.339573] Lustre: Skipped 1 previous similar message [11343.347266] Lustre: Skipped 1 previous similar message [11347.865289] Lustre: server umount lustre-OST0000 complete [11353.572197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.614156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11358.614866] Lustre: server umount lustre-MDT0000 complete [11358.691784] LustreError: 407500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.707445] LustreError: 407500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11362.121072] LustreError: 407484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320250 with bad export cookie 2572784338787895326 [11362.136634] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11362.416713] Lustre: server umount lustre-MDT0001 complete [11374.326610] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [11376.579319] Key type lgssc unregistered [11377.115823] LNet: 411232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11377.129492] LNetError: 411232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11378.180866] LNet: Removed LNI 192.168.203.112@tcp [11378.771708] Key type .llcrypt unregistered [11378.777450] Key type ._llcrypt unregistered [11406.609608] Key type ._llcrypt registered [11406.612284] Key type .llcrypt registered [11406.702927] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [11407.051505] Key type .llcrypt unregistered [11407.052762] Key type ._llcrypt unregistered [11426.658425] Key type ._llcrypt registered [11426.660569] Key type .llcrypt registered [11426.747734] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [11428.791378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11428.866395] alg: No test for adler32 (adler32-zlib) [11429.927973] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [11430.216758] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [11431.984435] Key type lgssc registered [11433.150682] Lustre: Echo OBD driver; http://www.lustre.org/ [11440.286493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11440.304379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.678692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11446.593801] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11451.871866] LustreError: 413449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.882257] LustreError: 413449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11454.535032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11454.863444] LustreError: 413450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11454.887152] LustreError: 413450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11454.907275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11460.066176] LustreError: 413449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.122954] LustreError: 413449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11461.093679] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11465.197056] LustreError: 413450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11465.215122] LustreError: 413450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11467.243260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11468.409227] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11472.357598] LustreError: 413444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.378282] LustreError: 413444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11476.640120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.961167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11482.027420] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11492.147808] Lustre: server umount lustre-OST0000 complete [11497.440667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11497.441212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11497.442907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11497.455065] Lustre: Skipped 1 previous similar message [11502.570334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11502.583459] Lustre: Skipped 2 previous similar messages [11505.669382] Lustre: server umount lustre-MDT0000 complete [11507.686524] Lustre: 412760:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786320380/real 1786320380] req@00000000413e31f5 x1873092613794304/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786320396 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11507.704523] LustreError: 413450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11507.733912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11507.776425] LustreError: 413450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11510.387433] LustreError: 415191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320398 with bad export cookie 16365306089244614551 [11510.388704] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11510.420669] LustreError: 415191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11510.902487] Lustre: server umount lustre-MDT0001 complete [11522.197494] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [11525.395552] Key type lgssc unregistered [11525.931840] LNet: 416598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11525.937410] LNetError: 416598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11525.966261] LNet: Removed LNI 192.168.203.112@tcp [11526.766243] Key type .llcrypt unregistered [11526.789149] Key type ._llcrypt unregistered [11551.306250] Key type ._llcrypt registered [11551.309137] Key type .llcrypt registered [11551.400318] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [11552.469296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11552.662338] alg: No test for adler32 (adler32-zlib) [11553.788697] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [11554.008369] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [11555.759233] Key type lgssc registered [11556.636293] Lustre: Echo OBD driver; http://www.lustre.org/ [11565.801235] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11565.818532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11581.373798] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11586.527337] LustreError: 417969:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.112@tcp: failed processing log, type 4: rc = -110 [11613.344512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11619.522572] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11627.444656] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11636.326283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11636.759689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11636.857871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11641.832431] LustreError: 417995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.849633] LustreError: 417995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11642.123536] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11646.953156] LustreError: 417994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11647.975107] LustreError: 417995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11648.002832] LustreError: 417995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11653.100130] LustreError: 419006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11654.328825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11654.744036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11659.539733] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11667.167550] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11674.413379] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11682.800241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11682.814447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11682.823404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11684.497342] Lustre: server umount lustre-OST0000 complete [11689.292371] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11689.346912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11689.359310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11690.469847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11690.485554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11695.578740] Lustre: server umount lustre-MDT0000 complete [11695.597081] LustreError: 419012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11695.638342] LustreError: 419012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11699.665270] LustreError: 417976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320588 with bad export cookie 9198222043951537426 [11699.670851] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11699.683348] LustreError: 417976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11710.180704] Lustre: server umount lustre-MDT0001 complete [11722.450216] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [11724.703729] Key type lgssc unregistered [11725.132866] LNet: 421547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11725.144354] LNetError: 421547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11725.163617] LNet: Removed LNI 192.168.203.112@tcp [11726.246839] Key type .llcrypt unregistered [11726.249610] Key type ._llcrypt unregistered [11754.355261] Key type ._llcrypt registered [11754.357681] Key type .llcrypt registered [11754.467715] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [11756.875357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11757.526663] alg: No test for adler32 (adler32-zlib) [11758.499751] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [11758.739189] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [11760.495168] Key type lgssc registered [11761.901358] Lustre: Echo OBD driver; http://www.lustre.org/ [11768.834459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11768.862706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11770.753741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11775.048234] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11781.088598] LustreError: 422964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11781.101445] LustreError: 422964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11783.008596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11783.244192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11787.324206] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11794.984131] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11800.441398] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11809.146185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11809.369453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11815.312168] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11815.555232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11823.420339] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11831.893301] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.12@tcp (stopping) [11832.479146] Lustre: server umount lustre-MDT0000 complete [11833.318357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11833.321893] LustreError: 423693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.360920] Lustre: Skipped 1 previous similar message [11833.439333] LustreError: 423693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11835.630360] LustreError: 422947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.12@tcp arrived at 1786320724 with bad export cookie 17107619006737747242 [11835.718910] LustreError: 422947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11837.857095] LustreError: 422945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320726 with bad export cookie 17107619006737746346 [11837.873628] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11838.436771] LustreError: 422965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11838.438667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.456592] LustreError: 422965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11838.478636] Lustre: Skipped 1 previous similar message [11838.484720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11840.163942] Lustre: server umount lustre-OST0000 complete [11842.466105] LustreError: 422964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.480315] LustreError: 422964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11847.777211] LustreError: 423695:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000879adda7 x1873092958781824/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11848.192944] Lustre: server umount lustre-MDT0001 complete [11859.264950] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [11861.461298] Key type lgssc unregistered [11861.744938] LNet: 426651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11861.752975] LNetError: 426651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11861.766886] LNet: Removed LNI 192.168.203.112@tcp [11862.458871] Key type .llcrypt unregistered [11862.462375] Key type ._llcrypt unregistered [11887.031628] Key type ._llcrypt registered [11887.033550] Key type .llcrypt registered [11887.128656] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [11888.995623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11889.082386] alg: No test for adler32 (adler32-zlib) [11890.084643] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [11890.269288] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [11891.931143] Key type lgssc registered [11893.078865] Lustre: Echo OBD driver; http://www.lustre.org/ [11900.462743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11900.524970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11901.997358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11906.206179] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11912.176471] LustreError: 428066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11912.235376] LustreError: 428066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11915.137410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11915.402758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11920.117825] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11927.195515] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11933.147546] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11939.892573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11940.269112] LustreError: 429886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11940.491145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11946.619353] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [11946.988559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11954.211853] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11961.840666] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11961.851058] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11961.879021] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11962.356081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11962.450623] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.12@tcp (stopping) [11962.466371] Lustre: Skipped 2 previous similar messages [11965.169431] Lustre: server umount lustre-MDT0001 complete [11967.461932] LustreError: 428066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11967.481727] LustreError: 428066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11972.578678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11972.578892] LustreError: 428793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.590266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11972.632801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11976.830553] Lustre: server umount lustre-OST0000 complete [11977.719613] LustreError: 428065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.215236] Lustre: server umount lustre-MDT0000 complete [11994.081902] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [11996.573856] Key type lgssc unregistered [11996.814620] LNet: 431752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11996.819641] LNetError: 431752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11996.827867] LNet: Removed LNI 192.168.203.112@tcp [11997.746364] Key type .llcrypt unregistered [11997.750209] Key type ._llcrypt unregistered [12021.968582] Key type ._llcrypt registered [12021.970784] Key type .llcrypt registered [12022.222560] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [12035.691447] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [12036.739491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12036.812052] alg: No test for adler32 (adler32-zlib) [12037.801512] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12038.064384] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [12039.711175] Key type lgssc registered [12040.786095] Lustre: Echo OBD driver; http://www.lustre.org/ [12076.465655] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [12091.621764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12091.656827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12092.909255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12092.945899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12093.038450] Lustre: lustre-MDT0000: new disk, initializing [12093.121116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12093.137904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12098.490981] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12115.019365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12115.288228] Lustre: lustre-OST0000: new disk, initializing [12115.294914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12115.300789] Lustre: Skipped 1 previous similar message [12115.307291] Lustre: 436777:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12115.384196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12121.542524] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12123.389801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12123.401178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12123.432388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12131.855639] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12149.400040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.493683] Lustre: 435761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12149.548657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12149.609630] Lustre: lustre-MDT0001: new disk, initializing [12149.698697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12149.744362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12149.791926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12153.944276] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12162.445688] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12162.665652] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12166.603928] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12166.903459] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12171.393576] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.12@tcp (stopping) [12175.331605] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12175.443775] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12175.480544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12175.502548] Lustre: Skipped 1 previous similar message [12177.369462] Lustre: server umount lustre-MDT0001 complete [12185.587159] LustreError: 435771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12185.622316] LustreError: 435771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12190.691147] LustreError: 437491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12190.706094] LustreError: 437491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12195.296310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12195.301313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.320510] Lustre: Skipped 1 previous similar message [12195.326188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12195.329829] Lustre: Skipped 1 previous similar message [12195.813598] LustreError: 435771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.581256] Lustre: server umount lustre-OST0000 complete [12200.930278] LustreError: 435772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12209.911420] Lustre: server umount lustre-MDT0000 complete [12223.367110] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [12225.552185] Key type lgssc unregistered [12225.756910] LNet: 440528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12225.761773] LNetError: 440528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12225.771806] LNet: Removed LNI 192.168.203.112@tcp [12226.624493] Key type .llcrypt unregistered [12226.626434] Key type ._llcrypt unregistered [12272.254811] Key type ._llcrypt registered [12272.257276] Key type .llcrypt registered [12272.364427] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [12272.637469] Key type .llcrypt unregistered [12272.639111] Key type ._llcrypt unregistered [12288.162364] Key type ._llcrypt registered [12288.165858] Key type .llcrypt registered [12288.342764] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [12289.898142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12289.953581] alg: No test for adler32 (adler32-zlib) [12291.015542] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [12291.208420] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [12292.849052] Key type lgssc registered [12293.964233] Lustre: Echo OBD driver; http://www.lustre.org/ [12322.319357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12322.344105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12323.545834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12323.716422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12323.762986] Lustre: lustre-MDT0000: new disk, initializing [12323.823588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12323.836366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12328.376853] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12337.912390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12338.064418] Lustre: 443910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12338.130418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12338.134881] Lustre: Skipped 1 previous similar message [12338.203969] Lustre: lustre-MDT0001: new disk, initializing [12338.384995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12338.420476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12338.437608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12342.866725] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12353.020790] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12358.456379] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12364.643741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12364.848720] Lustre: lustre-OST0000: new disk, initializing [12364.855692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12364.860757] Lustre: 445876:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12364.937383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12370.470487] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12372.192545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12372.202166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12372.306668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12380.802551] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12387.815119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.828857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12387.923252] Lustre: Skipped 2 previous similar messages [12387.931041] Lustre: Skipped 1 previous similar message [12392.929987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12392.934149] Lustre: Skipped 3 previous similar messages [12393.981910] Lustre: server umount lustre-MDT0000 complete [12398.362108] LustreError: 443902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786321286 with bad export cookie 1941269216520822779 [12398.371992] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12398.625175] Lustre: server umount lustre-MDT0001 complete [12411.695398] Lustre: server umount lustre-OST0000 complete [12425.140605] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:21:50 (1786321310) [12432.380500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12432.703525] LustreError: 447907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12432.736392] LustreError: 447907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12432.783917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12436.996481] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12437.985795] LustreError: 447908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12443.113131] LustreError: 447907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12444.213533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12444.534785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12449.303916] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12457.400902] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12463.726944] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12484.020690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12484.132150] Lustre: Found index 0 for lustre-OST0000, updating log [12484.282214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12489.875757] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [12496.540940] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12503.748804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12514.280282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12514.287854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12514.289078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12514.294042] Lustre: Skipped 1 previous similar message [12514.304024] Lustre: Skipped 1 previous similar message [12517.696443] Lustre: server umount lustre-OST0000 complete [12536.801083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12536.804568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12536.833508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12536.840664] Lustre: Skipped 2 previous similar messages [12541.931233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12541.943657] Lustre: Skipped 1 previous similar message [12542.559955] LNet: 1 local NIs in recovery (showing 1): 192.168.203.112@tcp [12548.063381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12548.210871] Lustre: server umount lustre-MDT0000 complete [12549.090523] LustreError: 447907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.124465] LustreError: 447907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12551.819960] LustreError: 447887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786321440 with bad export cookie 1941269216520824172 [12551.824544] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12551.838914] LustreError: 447887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12552.109988] Lustre: server umount lustre-MDT0001 complete [12560.492553] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:24:07 (1786321447) [13178.322118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13178.652466] LustreError: 454338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.705933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13183.407555] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13183.973506] LustreError: 454338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13189.098901] LustreError: 454339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13191.142686] LustreError: 454338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13191.203210] LustreError: 454338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13193.033317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13193.675517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13198.140989] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13206.323429] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13212.784639] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13233.311291] LNet: 1 local NIs in recovery (showing 1): 192.168.203.112@tcp [13233.382725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13233.510303] Lustre: Found index 0 for lustre-OST0000, updating log [13233.714156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13239.280729] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13246.650154] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13248.975619] Lustre: server umount lustre-OST0000 complete [13254.244397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13254.272357] Lustre: Skipped 1 previous similar message [13254.279642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13254.294392] Lustre: Skipped 5 previous similar messages [13255.343478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13255.349355] Lustre: Skipped 1 previous similar message [13257.376967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13257.380090] Lustre: Skipped 3 previous similar messages [13259.646742] Lustre: server umount lustre-MDT0000 complete [13260.525789] LustreError: 454338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13260.555454] LustreError: 454338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13264.143483] LustreError: 454321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786322152 with bad export cookie 1941269216520825257 [13264.147360] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13264.156100] LustreError: 454321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13264.332897] Lustre: server umount lustre-MDT0001 complete [13292.703550] LNet: 1 local NIs in recovery (showing 1): 192.168.203.112@tcp [13295.452304] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [13305.052246] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [13342.473936] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:37:08 (1786322228) [13361.928851] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [13364.677179] Key type lgssc unregistered [13364.880722] LNet: 460956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13364.887799] LNetError: 460956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13494.136822] LNet: Removed LNI 192.168.203.112@tcp [13494.753637] Key type .llcrypt unregistered [13494.755482] Key type ._llcrypt unregistered [13510.613335] Key type ._llcrypt registered [13510.615566] Key type .llcrypt registered [13510.693811] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [13512.556170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13512.578110] alg: No test for adler32 (adler32-zlib) [13513.631852] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [13513.865443] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [13515.575792] Key type lgssc registered [13516.568967] Lustre: Echo OBD driver; http://www.lustre.org/ [13520.679600] LNet: Added LNI 192.168.203.112@tcp100 [8/256/0/180] [13521.412773] LNet: Added LNI 192.168.203.112@tcp101 [8/256/0/180] [13522.006253] LNet: Added LNI 192.168.203.112@tcp102 [8/256/0/180] [13523.047446] LNet: Added LNI 192.168.203.112@tcp103 [8/256/0/180] [13524.035267] LNet: Added LNI 192.168.203.112@tcp104 [8/256/0/180] [13525.109235] LNet: Added LNI 192.168.203.112@tcp105 [8/256/0/180] [13526.353615] LNet: Added LNI 192.168.203.112@tcp106 [8/256/0/180] [13527.449371] LNet: Added LNI 192.168.203.112@tcp107 [8/256/0/180] [13528.543026] LNet: Added LNI 192.168.203.112@tcp108 [8/256/0/180] [13529.743756] LNet: Added LNI 192.168.203.112@tcp109 [8/256/0/180] [13530.639808] LNet: Added LNI 192.168.203.112@tcp110 [8/256/0/180] [13531.183616] LNet: Added LNI 192.168.203.112@tcp111 [8/256/0/180] [13531.929852] LNet: Added LNI 192.168.203.112@tcp112 [8/256/0/180] [13532.597320] LNet: Added LNI 192.168.203.112@tcp113 [8/256/0/180] [13533.415753] LNet: Added LNI 192.168.203.112@tcp114 [8/256/0/180] [13534.415672] LNet: Added LNI 192.168.203.112@tcp115 [8/256/0/180] [13535.143382] LNet: Added LNI 192.168.203.112@tcp116 [8/256/0/180] [13536.146479] LNet: Added LNI 192.168.203.112@tcp117 [8/256/0/180] [13536.886385] LNet: Added LNI 192.168.203.112@tcp118 [8/256/0/180] [13537.963069] LNet: Added LNI 192.168.203.112@tcp119 [8/256/0/180] [13539.023153] LNet: Added LNI 192.168.203.112@tcp120 [8/256/0/180] [13539.896663] LNet: Added LNI 192.168.203.112@tcp121 [8/256/0/180] [13540.871640] LNet: Added LNI 192.168.203.112@tcp122 [8/256/0/180] [13542.025265] LNet: Added LNI 192.168.203.112@tcp123 [8/256/0/180] [13543.195592] LNet: Added LNI 192.168.203.112@tcp124 [8/256/0/180] [13544.274206] LNet: Added LNI 192.168.203.112@tcp125 [8/256/0/180] [13545.270461] LNet: Added LNI 192.168.203.112@tcp126 [8/256/0/180] [13546.153435] LNet: Added LNI 192.168.203.112@tcp127 [8/256/0/180] [13547.096338] LNet: Added LNI 192.168.203.112@tcp128 [8/256/0/180] [13547.740635] LNet: Added LNI 192.168.203.112@tcp129 [8/256/0/180] [13548.417798] LNet: Added LNI 192.168.203.112@tcp130 [8/256/0/180] [13549.319882] LNet: Added LNI 192.168.203.112@tcp131 [8/256/0/180] [13549.977548] LNet: Added LNI 192.168.203.112@tcp132 [8/256/0/180] [13550.740964] LNet: Added LNI 192.168.203.112@tcp133 [8/256/0/180] [13551.400891] LNet: Added LNI 192.168.203.112@tcp134 [8/256/0/180] [13552.160058] LNet: Added LNI 192.168.203.112@tcp135 [8/256/0/180] [13559.291558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13559.343209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13560.418890] Lustre: 463989:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.112@tcp: can't add NID 192.168.203.112@tcp130: rc = -75 [13560.487898] LNetError: 461448:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.112 [13560.499784] LNetError: Protocol error connecting to 192.168.203.112@tcp113 at host 192.168.203.112:988: is it running a compatible version of Lustre? [13560.511930] LNetError: Skipped 8 previous similar messages [13560.580636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13560.600714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13560.666189] Lustre: lustre-MDT0000: new disk, initializing [13560.727462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13560.744684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13560.994873] LNet: 9 local NIs in recovery (showing 9): 192.168.203.112@tcp121, 192.168.203.112@tcp123, 192.168.203.112@tcp124, 192.168.203.112@tcp125, 192.168.203.112@tcp126, 192.168.203.112@tcp128, 192.168.203.112@tcp127, 192.168.203.112@tcp129, 192.168.203.112@tcp119 [13562.080213] LNetError: 464014:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.112 [13562.094425] LNetError: 464014:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [13562.101607] LNetError: Protocol error connecting to 192.168.203.112@tcp121 at host 192.168.203.112:988: is it running a compatible version of Lustre? [13562.113399] LNetError: Skipped 8 previous similar messages [13564.130587] LNetError: 464021:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.112 [13564.140433] LNetError: 464021:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [13564.154714] LNetError: Protocol error connecting to 192.168.203.112@tcp121 at host 192.168.203.112:988: is it running a compatible version of Lustre? [13564.769153] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13568.230129] LNetError: 461451:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.112 [13568.237206] LNetError: Protocol error connecting to 192.168.203.112@tcp121 at host 192.168.203.112:988: is it running a compatible version of Lustre? [13574.248854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13574.406529] Lustre: 464005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13574.480164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13574.490758] Lustre: Skipped 1 previous similar message [13574.575119] Lustre: lustre-MDT0001: new disk, initializing [13574.644226] Lustre: 464807:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.112@tcp: can't add NID 192.168.203.112@tcp130: rc = -75 [13574.649592] Lustre: 464807:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13574.820868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13574.871218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13574.886041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13576.056414] Lustre: 464884:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.112@tcp: can't add NID 192.168.203.112@tcp130: rc = -75 [13576.061401] Lustre: 464884:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13576.429215] LNetError: 464014:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.112 [13576.446527] LNetError: Protocol error connecting to 192.168.203.112@tcp121 at host 192.168.203.112:988: is it running a compatible version of Lustre? [13579.823346] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13589.831684] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13591.712589] LNetError: 461448:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.112 [13591.742672] LNetError: Protocol error connecting to 192.168.203.112@tcp120 at host 192.168.203.112:988: is it running a compatible version of Lustre? [13597.907901] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13604.891367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13605.122984] Lustre: lustre-OST0000: new disk, initializing [13605.126510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13605.131894] Lustre: 465979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13605.214882] Lustre: 465984:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.112@tcp: can't add NID 192.168.203.112@tcp130: rc = -75 [13605.226979] Lustre: 465984:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13605.276503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13608.103097] LNetError: 461448:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.112 [13608.111721] LNetError: 461448:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [13608.116533] LNetError: Protocol error connecting to 192.168.203.112@tcp113 at host 192.168.203.112:988: is it running a compatible version of Lustre? [13608.122523] LNetError: Skipped 8 previous similar messages [13610.526295] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13613.076814] Lustre: 466436:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.112@tcp: can't add NID 192.168.203.112@tcp130: rc = -75 [13613.090171] Lustre: 466436:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13613.150214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13613.156258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13613.255353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13620.639223] LNet: 1 local NIs in recovery (showing 1): 192.168.203.112@tcp121 [13622.550606] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13634.528414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13634.531783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13634.536909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13634.548351] Lustre: Skipped 1 previous similar message [13635.790965] Lustre: server umount lustre-OST0000 complete [13639.648884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13639.670125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13639.683702] Lustre: Skipped 2 previous similar messages [13644.768923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13645.488575] Lustre: server umount lustre-MDT0000 complete [13648.501518] LustreError: 467136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786322537 with bad export cookie 3487310593171802944 [13648.509267] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13648.512341] LustreError: 467136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13648.774504] Lustre: server umount lustre-MDT0001 complete [13655.585537] LNetError: 461451:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.112 [13655.590733] LNetError: 461451:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [13655.594803] LNetError: Protocol error connecting to 192.168.203.112@tcp120 at host 192.168.203.112:988: is it running a compatible version of Lustre? [13655.600453] LNetError: Skipped 13 previous similar messages [13659.301758] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [13661.588709] Key type lgssc unregistered [13661.845870] LNet: 467898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13661.852127] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13662.879162] LNet: Removed LNI 192.168.203.112@tcp [13662.885374] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13662.897377] LNet: Removed LNI 192.168.203.112@tcp100 [13663.967147] LNet: Removed LNI 192.168.203.112@tcp101 [13663.973203] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13663.982185] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13663.995114] LNet: Removed LNI 192.168.203.112@tcp102 [13665.094819] LNet: Removed LNI 192.168.203.112@tcp103 [13666.143149] LNet: Removed LNI 192.168.203.112@tcp104 [13666.146235] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13666.151572] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13666.156344] LNet: Removed LNI 192.168.203.112@tcp105 [13667.167165] LNet: Removed LNI 192.168.203.112@tcp106 [13668.191338] LNet: Removed LNI 192.168.203.112@tcp107 [13669.215285] LNet: Removed LNI 192.168.203.112@tcp108 [13670.239170] LNet: Removed LNI 192.168.203.112@tcp109 [13670.245457] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13670.251658] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13670.256330] LNet: Removed LNI 192.168.203.112@tcp110 [13671.290819] LNet: Removed LNI 192.168.203.112@tcp111 [13672.351657] LNet: Removed LNI 192.168.203.112@tcp112 [13673.375181] LNet: Removed LNI 192.168.203.112@tcp113 [13674.399353] LNet: Removed LNI 192.168.203.112@tcp114 [13675.423151] LNet: Removed LNI 192.168.203.112@tcp115 [13676.455804] LNet: Removed LNI 192.168.203.112@tcp116 [13677.471420] LNet: Removed LNI 192.168.203.112@tcp117 [13678.495174] LNet: Removed LNI 192.168.203.112@tcp118 [13678.502089] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13678.518859] LNetError: 467898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13678.531599] LNet: Removed LNI 192.168.203.112@tcp119 [13679.583150] LNet: Removed LNI 192.168.203.112@tcp120 [13679.585529] LNet: Removed LNI 192.168.203.112@tcp121 [13680.616620] LNet: Removed LNI 192.168.203.112@tcp122 [13681.633716] LNet: Removed LNI 192.168.203.112@tcp123 [13682.668760] LNet: Removed LNI 192.168.203.112@tcp124 [13683.679218] LNet: Removed LNI 192.168.203.112@tcp125 [13684.703451] LNet: Removed LNI 192.168.203.112@tcp126 [13685.727194] LNet: Removed LNI 192.168.203.112@tcp127 [13686.751284] LNet: Removed LNI 192.168.203.112@tcp128 [13687.775190] LNet: Removed LNI 192.168.203.112@tcp129 [13687.780822] LNet: Removed LNI 192.168.203.112@tcp130 [13687.785722] LNet: Removed LNI 192.168.203.112@tcp131 [13687.791526] LNet: Removed LNI 192.168.203.112@tcp132 [13687.796623] LNet: Removed LNI 192.168.203.112@tcp133 [13687.801977] LNet: Removed LNI 192.168.203.112@tcp134 [13687.812828] LNet: Removed LNI 192.168.203.112@tcp135 [13688.679217] Key type .llcrypt unregistered [13688.687790] Key type ._llcrypt unregistered [13712.236139] Key type ._llcrypt registered [13712.237962] Key type .llcrypt registered [13712.331815] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [13726.499183] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [13728.699088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13728.870828] alg: No test for adler32 (adler32-zlib) [13730.188296] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [13730.428478] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [13732.104190] Key type lgssc registered [13733.224918] Lustre: Echo OBD driver; http://www.lustre.org/ [13768.357746] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:44:14 (1786322654) [13784.670603] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [13787.462569] Key type lgssc unregistered [13787.753765] LNet: 471991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13787.759937] LNetError: 471991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13787.775939] LNet: Removed LNI 192.168.203.112@tcp [13788.521594] Key type .llcrypt unregistered [13788.526519] Key type ._llcrypt unregistered [13803.612729] Key type ._llcrypt registered [13803.615408] Key type .llcrypt registered [13803.703380] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [13805.217167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13805.244580] alg: No test for adler32 (adler32-zlib) [13806.312730] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [13806.503587] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [13808.135431] Key type lgssc registered [13809.089801] Lustre: Echo OBD driver; http://www.lustre.org/ [13817.164445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13817.209941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13818.362665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13818.388436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13818.456274] Lustre: lustre-MDT0000: new disk, initializing [13818.522299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13818.535813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13822.585536] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13831.752294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13831.835119] Lustre: 473199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13831.885863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13831.889657] Lustre: Skipped 1 previous similar message [13831.938986] Lustre: lustre-MDT0001: new disk, initializing [13832.020284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13832.041574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13832.051789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13835.825448] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13844.674282] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13849.820361] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13856.215400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13856.437320] Lustre: lustre-OST0000: new disk, initializing [13856.441330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13856.447669] Lustre: 475166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13856.541124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13861.851800] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [13866.008243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13866.014527] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13866.061166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13870.714313] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13874.260409] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing lnet_if_list [13879.346829] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13879.386463] LNet: Added LNI 192.168.203.112@tcp10 [8/256/0/180] [13884.762630] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13884.786470] LNet: Added LNI 192.168.203.112@tcp11 [8/256/0/180] [13888.761406] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13888.800586] LNet: Added LNI 192.168.203.112@tcp12 [8/256/0/180] [13893.048959] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13893.078386] LNet: Added LNI 192.168.203.112@tcp13 [8/256/0/180] [13896.945771] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13896.969040] LNet: Added LNI 192.168.203.112@tcp14 [8/256/0/180] [13901.140912] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13901.184461] LNet: Added LNI 192.168.203.112@tcp15 [8/256/0/180] [13905.805310] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13905.828774] LNet: Added LNI 192.168.203.112@tcp16 [8/256/0/180] [13910.906434] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13910.992716] LNet: Added LNI 192.168.203.112@tcp17 [8/256/0/180] [13915.420775] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13915.441804] LNet: Added LNI 192.168.203.112@tcp18 [8/256/0/180] [13920.142360] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13920.187437] LNet: Added LNI 192.168.203.112@tcp19 [8/256/0/180] [13925.121251] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13925.149967] LNet: Added LNI 192.168.203.112@tcp20 [8/256/0/180] [13929.938470] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13929.959176] LNet: Added LNI 192.168.203.112@tcp21 [8/256/0/180] [13934.436598] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13934.459427] LNet: Added LNI 192.168.203.112@tcp22 [8/256/0/180] [13939.445835] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13939.570498] LNet: Added LNI 192.168.203.112@tcp23 [8/256/0/180] [13945.863451] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13945.880854] LNet: Added LNI 192.168.203.112@tcp24 [8/256/0/180] [13951.222193] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13951.254252] LNet: Added LNI 192.168.203.112@tcp25 [8/256/0/180] [13956.369596] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13956.406411] LNet: Added LNI 192.168.203.112@tcp26 [8/256/0/180] [13963.141977] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13963.163990] LNet: Added LNI 192.168.203.112@tcp27 [8/256/0/180] [13969.627237] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13969.657521] LNet: Added LNI 192.168.203.112@tcp28 [8/256/0/180] [13975.498853] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13975.520769] LNet: Added LNI 192.168.203.112@tcp29 [8/256/0/180] [13981.484682] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13981.504152] LNet: Added LNI 192.168.203.112@tcp30 [8/256/0/180] [13987.423926] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13987.447405] LNet: Added LNI 192.168.203.112@tcp31 [8/256/0/180] [13993.855175] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13993.926875] LNet: Added LNI 192.168.203.112@tcp32 [8/256/0/180] [14000.593810] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14000.618038] LNet: Added LNI 192.168.203.112@tcp33 [8/256/0/180] [14007.802725] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14007.823843] LNet: Added LNI 192.168.203.112@tcp34 [8/256/0/180] [14014.498832] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14014.601200] LNet: Added LNI 192.168.203.112@tcp35 [8/256/0/180] [14021.061931] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14021.090318] LNet: Added LNI 192.168.203.112@tcp36 [8/256/0/180] [14028.086394] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14028.110249] LNet: Added LNI 192.168.203.112@tcp37 [8/256/0/180] [14035.583157] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14035.603290] LNet: Added LNI 192.168.203.112@tcp38 [8/256/0/180] [14043.063529] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14043.103898] LNet: Added LNI 192.168.203.112@tcp39 [8/256/0/180] [14050.417265] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14050.471336] LNet: Added LNI 192.168.203.112@tcp40 [8/256/0/180] [14056.542676] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14056.564614] LNet: Added LNI 192.168.203.112@tcp41 [8/256/0/180] [14064.270335] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14064.373686] LNet: Added LNI 192.168.203.112@tcp42 [8/256/0/180] [14072.717513] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14072.741461] LNet: Added LNI 192.168.203.112@tcp43 [8/256/0/180] [14081.552568] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14081.588530] LNet: Added LNI 192.168.203.112@tcp44 [8/256/0/180] [14089.167319] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14089.187703] LNet: Added LNI 192.168.203.112@tcp45 [8/256/0/180] [14097.560297] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14097.579545] LNet: Added LNI 192.168.203.112@tcp46 [8/256/0/180] [14105.437852] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14105.466158] LNet: Added LNI 192.168.203.112@tcp47 [8/256/0/180] [14113.401351] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14113.423208] LNet: Added LNI 192.168.203.112@tcp48 [8/256/0/180] [14120.935706] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14120.954742] LNet: Added LNI 192.168.203.112@tcp49 [8/256/0/180] [14129.873785] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14129.908691] LNet: Added LNI 192.168.203.112@tcp50 [8/256/0/180] [14149.603492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14149.621457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14149.636510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14152.680245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14152.681130] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14152.693313] Lustre: Skipped 1 previous similar message [14152.723739] Lustre: Skipped 1 previous similar message [14154.943490] Lustre: server umount lustre-MDT0000 complete [14158.684353] LustreError: 473189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786323047 with bad export cookie 16464959727890016011 [14158.691302] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14158.694937] LustreError: 473189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14159.029239] Lustre: server umount lustre-MDT0001 complete [14175.913751] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [14179.301499] Lustre: 472471:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786323051/real 1786323051] req@000000004ada044c x1873095106397824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786323067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14179.381306] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14180.322741] Lustre: 472472:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786323052/real 1786323052] req@000000009ff31c71 x1873095106397952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786323068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14184.417858] Lustre: 472469:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786323056/real 1786323056] req@000000006e0fb523 x1873095106398080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786323072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14184.530107] Lustre: 472469:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14186.956952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14189.535239] Lustre: 472472:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786323062/real 1786323062] req@0000000036c87807 x1873095106398464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786323078 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14194.669593] Lustre: 472472:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786323067/real 1786323067] req@00000000643af080 x1873095106398592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786323083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14204.129309] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe47f49a32dff30ff [14204.137663] Lustre: MGC192.168.203.112@tcp: Connection restored to 0@lo (at 0@lo) [14204.315700] LustreError: 475159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14204.332528] LustreError: 475159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14204.364582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14205.461905] LustreError: 499400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14205.480404] LustreError: 499400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14207.526024] LustreError: 475160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14207.534894] LustreError: 475160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14207.539992] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14208.647811] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14212.591240] LustreError: 499400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14212.634915] LustreError: 499400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14216.594241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14216.864620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14220.987363] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14222.331984] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14228.288216] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14235.144202] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14255.587493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14255.591524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14255.598270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14255.603177] Lustre: Skipped 1 previous similar message [14258.147412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14258.158808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14261.575749] Lustre: server umount lustre-OST0000 complete [14268.387644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14268.387983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14268.390231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14268.390239] Lustre: Skipped 1 previous similar message [14268.394798] Lustre: Skipped 2 previous similar messages [14277.601147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14277.605339] Lustre: Skipped 4 previous similar messages [14280.159115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14280.313675] Lustre: server umount lustre-MDT0000 complete [14282.729928] LustreError: 499401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14282.729928] LustreError: 499405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14284.572112] LustreError: 473190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786323173 with bad export cookie 16464959727890018559 [14284.576746] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14284.583533] LustreError: 473190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14284.903160] Lustre: server umount lustre-MDT0001 complete [14294.778853] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [14296.961929] Key type lgssc unregistered [14297.304795] LNet: 502207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14297.323631] LNetError: 502207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14297.330710] LNet: Removed LNI 192.168.203.112@tcp [14297.340480] LNet: Removed LNI 192.168.203.112@tcp10 [14297.350442] LNet: Removed LNI 192.168.203.112@tcp11 [14297.354540] LNet: Removed LNI 192.168.203.112@tcp12 [14297.358013] LNet: Removed LNI 192.168.203.112@tcp13 [14297.361597] LNet: Removed LNI 192.168.203.112@tcp14 [14297.364862] LNet: Removed LNI 192.168.203.112@tcp15 [14297.367953] LNet: Removed LNI 192.168.203.112@tcp16 [14297.371538] LNet: Removed LNI 192.168.203.112@tcp17 [14297.374307] LNet: Removed LNI 192.168.203.112@tcp18 [14297.376617] LNet: Removed LNI 192.168.203.112@tcp19 [14297.380632] LNet: Removed LNI 192.168.203.112@tcp20 [14297.383881] LNet: Removed LNI 192.168.203.112@tcp21 [14297.390481] LNet: Removed LNI 192.168.203.112@tcp22 [14297.396177] LNet: Removed LNI 192.168.203.112@tcp23 [14297.408935] LNet: Removed LNI 192.168.203.112@tcp24 [14297.412917] LNet: Removed LNI 192.168.203.112@tcp25 [14297.415722] LNet: Removed LNI 192.168.203.112@tcp26 [14297.418104] LNet: Removed LNI 192.168.203.112@tcp27 [14297.420938] LNet: Removed LNI 192.168.203.112@tcp28 [14297.423849] LNet: Removed LNI 192.168.203.112@tcp29 [14297.426938] LNet: Removed LNI 192.168.203.112@tcp30 [14297.430231] LNet: Removed LNI 192.168.203.112@tcp31 [14297.433780] LNet: Removed LNI 192.168.203.112@tcp32 [14297.436649] LNet: Removed LNI 192.168.203.112@tcp33 [14297.439964] LNet: Removed LNI 192.168.203.112@tcp34 [14297.443094] LNet: Removed LNI 192.168.203.112@tcp35 [14297.446327] LNet: Removed LNI 192.168.203.112@tcp36 [14297.449660] LNet: Removed LNI 192.168.203.112@tcp37 [14297.452658] LNet: Removed LNI 192.168.203.112@tcp38 [14297.455671] LNet: Removed LNI 192.168.203.112@tcp39 [14297.458859] LNet: Removed LNI 192.168.203.112@tcp40 [14297.461886] LNet: Removed LNI 192.168.203.112@tcp41 [14297.464882] LNet: Removed LNI 192.168.203.112@tcp42 [14297.467884] LNet: Removed LNI 192.168.203.112@tcp43 [14297.470966] LNet: Removed LNI 192.168.203.112@tcp44 [14297.473929] LNet: Removed LNI 192.168.203.112@tcp45 [14297.476632] LNet: Removed LNI 192.168.203.112@tcp46 [14297.479881] LNet: Removed LNI 192.168.203.112@tcp47 [14297.483643] LNet: Removed LNI 192.168.203.112@tcp48 [14297.486691] LNet: Removed LNI 192.168.203.112@tcp49 [14297.496954] LNet: Removed LNI 192.168.203.112@tcp50 [14298.249455] Key type .llcrypt unregistered [14298.251213] Key type ._llcrypt unregistered [14318.183870] Key type ._llcrypt registered [14318.185768] Key type .llcrypt registered [14318.297376] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [14331.012912] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [14332.530524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14332.904060] alg: No test for adler32 (adler32-zlib) [14333.951340] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [14334.227685] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [14336.039132] Key type lgssc registered [14337.073976] Lustre: Echo OBD driver; http://www.lustre.org/ [14370.383659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14370.406995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14371.681714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14371.797454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14371.888733] Lustre: lustre-MDT0000: new disk, initializing [14371.961615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14371.976561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14376.916942] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14387.663513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14387.819392] Lustre: 505385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14387.921854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14387.934805] Lustre: Skipped 1 previous similar message [14388.018770] Lustre: lustre-MDT0001: new disk, initializing [14388.127993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14388.194858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14388.200767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14392.742606] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14401.710934] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14407.639879] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14416.237815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14416.452706] Lustre: *** cfs_fail_loc=90f, val=0*** [14416.528741] Lustre: lustre-OST0000: new disk, initializing [14416.533070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14416.559687] Lustre: 507402:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14416.704625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14421.956934] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14422.584258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14422.597822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14422.676721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14431.198837] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14438.373123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14438.374671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14438.378916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14438.381534] LustreError: Skipped 1 previous similar message [14438.431971] Lustre: Skipped 1 previous similar message [14443.489986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14443.496277] Lustre: Skipped 2 previous similar messages [14444.517113] Lustre: server umount lustre-OST0000 complete [14448.611620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14448.615532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14448.636446] Lustre: Skipped 1 previous similar message [14448.670914] Lustre: Skipped 1 previous similar message [14453.739714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14453.759164] Lustre: Skipped 1 previous similar message [14453.915165] Lustre: server umount lustre-MDT0000 complete [14458.091295] LustreError: 505376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786323346 with bad export cookie 15340618453030351903 [14458.096536] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14458.104782] LustreError: 505376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14458.661799] Lustre: server umount lustre-MDT0001 complete [14469.622867] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [14471.841703] Key type lgssc unregistered [14472.078658] LNet: 509308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14472.083683] LNetError: 509308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14473.125570] LNet: Removed LNI 192.168.203.112@tcp [14473.668302] Key type .llcrypt unregistered [14473.670275] Key type ._llcrypt unregistered [14492.427478] Key type ._llcrypt registered [14492.429610] Key type .llcrypt registered [14492.509948] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [14494.269317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14494.304863] alg: No test for adler32 (adler32-zlib) [14495.743594] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [14495.936423] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [14497.628540] Key type lgssc registered [14498.654231] Lustre: Echo OBD driver; http://www.lustre.org/ [14505.633302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14505.648274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14507.278841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14511.451235] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14516.710890] LustreError: 510580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14516.724335] LustreError: 510580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14518.578878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14518.907304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14522.865347] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14529.566435] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14534.324717] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14541.146170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14541.487337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14548.223297] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14556.102458] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14565.141782] loop0: detected capacity change from 0 to 204800000 [14565.375511] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14565.381711] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14565.390399] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14570.559156] loop0: detected capacity change from 0 to 204800000 [14570.682169] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14570.696665] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14570.705911] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14575.396606] loop0: detected capacity change from 0 to 204800000 [14579.402222] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14579.495983] Lustre: fs15246-OST0000: new disk, initializing [14579.499338] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14579.504654] Lustre: 514171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14579.540769] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14584.426062] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14591.879546] loop1: detected capacity change from 0 to 204800000 [14596.854375] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14596.936064] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14597.064988] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14597.281289] Lustre: fs15246-MDT0000: new disk, initializing [14597.506748] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14597.535853] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14597.642358] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14598.694753] Lustre: 515291:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007da8af61 x1873095828744320/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786323503 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14601.184984] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14607.786346] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14607.791786] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14607.809695] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14621.673274] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14621.682296] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14623.604212] Lustre: server umount fs15246-MDT0000 complete [14626.792271] LustreError: 515649:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14631.335960] Lustre: server umount fs15246-OST0000 complete [14641.159714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14641.172288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14641.205126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14646.866378] Lustre: server umount lustre-OST0000 complete [14651.874603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14651.886702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14651.910515] Lustre: Skipped 1 previous similar message [14651.918971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14651.927477] Lustre: Skipped 2 previous similar messages [14657.508392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14657.515195] Lustre: Skipped 3 previous similar messages [14662.624403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14662.630246] Lustre: Skipped 1 previous similar message [14664.159199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14664.432069] Lustre: server umount lustre-MDT0000 complete [14666.744511] LustreError: 510574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14666.780488] LustreError: 510574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14668.443317] LustreError: 510559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786323557 with bad export cookie 12503716055487177092 [14668.451473] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14668.458546] LustreError: 510559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14668.911374] Lustre: server umount lustre-MDT0001 complete [14679.411508] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [14682.267879] Key type lgssc unregistered [14682.728780] LNet: 518391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14682.733718] LNetError: 518391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14682.746186] LNet: Removed LNI 192.168.203.112@tcp [14683.718152] Key type .llcrypt unregistered [14683.720221] Key type ._llcrypt unregistered [14703.283775] Key type ._llcrypt registered [14703.293302] Key type .llcrypt registered [14703.388840] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [14704.919738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14705.038615] alg: No test for adler32 (adler32-zlib) [14705.998514] Lustre: Lustre: Build Version: 2.17.56_55_g704561f [14706.166881] LNet: Added LNI 192.168.203.112@tcp [8/256/0/180] [14707.815173] Key type lgssc registered [14708.828647] Lustre: Echo OBD driver; http://www.lustre.org/ [14742.308216] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [14750.986750] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [14780.504661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14780.531862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14781.799940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14781.819895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14781.869272] Lustre: lustre-MDT0000: new disk, initializing [14781.922573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14781.936581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14785.755601] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14795.118315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14795.255436] Lustre: 522872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14795.308554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14795.312623] Lustre: Skipped 1 previous similar message [14795.377332] Lustre: lustre-MDT0001: new disk, initializing [14795.462134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14795.498800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14795.515045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14799.934951] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14809.479350] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14815.501636] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14822.466409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14822.738992] Lustre: lustre-OST0000: new disk, initializing [14822.742135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14822.746977] Lustre: 524838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14822.822691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14828.630735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14828.647361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14828.771116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14829.009924] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug -1 all [14838.809844] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14843.620678] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14843.861636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14848.157465] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14848.418863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14854.625980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14854.636693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14854.712444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14859.747880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14859.752727] Lustre: Skipped 2 previous similar messages [14864.351255] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14864.441068] Lustre: server umount lustre-OST0000 complete [14869.984950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14869.987758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14869.990029] Lustre: Skipped 2 previous similar messages [14870.001613] Lustre: Skipped 1 previous similar message [14875.105267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14875.109984] Lustre: Skipped 1 previous similar message [14880.224348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14880.228852] Lustre: Skipped 1 previous similar message [14882.271633] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14882.426117] Lustre: server umount lustre-MDT0000 complete [14885.620728] LustreError: 522863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786323774 with bad export cookie 12717398020075665161 [14885.632710] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14885.651264] LustreError: 522863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14885.867363] Lustre: server umount lustre-MDT0001 complete [14908.472430] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid [14918.450711] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local [14950.476515] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14676 sec ========= 21:03:56 (1786323836) [14952.433374] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:03:58 (1786323838) === [14956.413466] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:04:02 (1786323842) === [14992.159440] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing unload_modules_local [14994.153409] Key type lgssc unregistered [14994.472071] LNet: 531487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14994.475776] LNetError: 531487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14995.497754] LNet: Removed LNI 192.168.203.112@tcp [14997.024218] Key type .llcrypt unregistered [14997.027372] Key type ._llcrypt unregistered