[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489341954 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002201] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006335] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009112] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012064] *** VALIDATE selinux *** [ 0.018557] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022665] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025099] *** VALIDATE tmpfs *** [ 0.026442] *** VALIDATE proc *** [ 0.028292] *** VALIDATE cgroup *** [ 0.029008] *** VALIDATE cgroup2 *** [ 0.030260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032051] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034028] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.037795] debug: unmapping init [mem 0xffffffffb4259000-0xffffffffb4260fff] [ 0.039775] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040639] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041017] ... version: 2 [ 0.042000] ... bit width: 48 [ 0.042771] ... generic registers: 4 [ 0.043007] ... value mask: 0000ffffffffffff [ 0.044007] ... max period: 00007fffffffffff [ 0.045007] ... fixed-purpose events: 3 [ 0.046006] ... event mask: 000000070000000f [ 0.047272] rcu: Hierarchical SRCU implementation. [ 0.049921] smp: Bringing up secondary CPUs ... [ 0.050543] x86: Booting SMP configuration: [ 0.051019] .... node #0, CPUs: #1 #2 #3 [ 0.060665] smp: Brought up 1 node, 4 CPUs [ 0.062021] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.154146] node 0 deferred pages initialised in 86ms [ 0.158198] devtmpfs: initialized [ 0.159291] x86/mm: Memory block size: 128MB [ 0.161975] gcov: version magic: 0x41383552 [ 0.163404] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.170445] pinctrl core: initialized pinctrl subsystem [ 0.174227] [ 0.174699] ************************************************************* [ 0.176014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178011] ** ** [ 0.180027] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.182015] ** ** [ 0.184018] ** This means that this kernel is built to expose internal ** [ 0.185000] ** IOMMU data structures, which may compromise security on ** [ 0.186018] ** your system. ** [ 0.189020] ** ** [ 0.191017] ** If you see this message and you are not debugging the ** [ 0.194018] ** kernel, report this immediately to your vendor! ** [ 0.196015] ** ** [ 0.198015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.201018] ************************************************************* [ 0.204942] NET: Registered protocol family 16 [ 0.207372] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.216220] cpuidle: using governor menu [ 0.217760] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.220695] PCI: Using configuration type 1 for base access [ 0.223159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235692] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238128] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.242207] cryptd: max_cpu_qlen set to 1000 [ 0.246209] ACPI: Added _OSI(Module Device) [ 0.248014] ACPI: Added _OSI(Processor Device) [ 0.250016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.252021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.264954] ACPI: Interpreter enabled [ 0.266236] ACPI: PM: (supports S0 S3 S4 S5) [ 0.271027] ACPI: Using IOAPIC for interrupt routing [ 0.274194] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.279547] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.289431] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.291055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.297149] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.307062] acpiphp: Slot [2] registered [ 0.309801] acpiphp: Slot [5] registered [ 0.312218] acpiphp: Slot [6] registered [ 0.315212] acpiphp: Slot [7] registered [ 0.318299] acpiphp: Slot [8] registered [ 0.319149] acpiphp: Slot [9] registered [ 0.322309] acpiphp: Slot [10] registered [ 0.324207] acpiphp: Slot [3] registered [ 0.326160] acpiphp: Slot [4] registered [ 0.328122] acpiphp: Slot [11] registered [ 0.329118] acpiphp: Slot [12] registered [ 0.331093] acpiphp: Slot [13] registered [ 0.332114] acpiphp: Slot [14] registered [ 0.334108] acpiphp: Slot [15] registered [ 0.335105] acpiphp: Slot [16] registered [ 0.337118] acpiphp: Slot [17] registered [ 0.338115] acpiphp: Slot [18] registered [ 0.340086] acpiphp: Slot [19] registered [ 0.341080] acpiphp: Slot [20] registered [ 0.342082] acpiphp: Slot [21] registered [ 0.343079] acpiphp: Slot [22] registered [ 0.345071] acpiphp: Slot [23] registered [ 0.346078] acpiphp: Slot [24] registered [ 0.347125] acpiphp: Slot [25] registered [ 0.348074] acpiphp: Slot [26] registered [ 0.349085] acpiphp: Slot [27] registered [ 0.350111] acpiphp: Slot [28] registered [ 0.352081] acpiphp: Slot [29] registered [ 0.353078] acpiphp: Slot [30] registered [ 0.354110] acpiphp: Slot [31] registered [ 0.355060] PCI host bridge to bus 0000:00 [ 0.356013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.358022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.360022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.362033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.366028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.368236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.370014] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.373237] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.382017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.386053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.388020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.390016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.392014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.395457] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.399925] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.402057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.406000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.411025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.422834] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.429033] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.436757] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.458028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.487000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.503023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.541024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.550872] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.562023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.571020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.590017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.613699] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.630024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.638026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.666025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.680823] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.696839] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.708022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.737025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.754804] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.766025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.773027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.796026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.812304] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.815502] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.819500] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.821334] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.824200] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.832725] iommu: Default domain type: Passthrough [ 0.833472] SCSI subsystem initialized [ 0.834171] ACPI: bus type USB registered [ 0.835159] usbcore: registered new interface driver usbfs [ 0.838298] usbcore: registered new interface driver hub [ 0.841112] usbcore: registered new device driver usb [ 0.845438] pps_core: LinuxPPS API ver. 1 registered [ 0.849076] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.858104] PTP clock support registered [ 0.867041] EDAC MC: Ver: 3.0.0 [ 0.868401] PCI: Using ACPI for IRQ routing [ 0.872528] NetLabel: Initializing [ 0.874229] NetLabel: domain hash size = 128 [ 0.878115] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.883109] NetLabel: unlabeled traffic allowed by default [ 0.885147] vgaarb: loaded [ 0.888027] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.889015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.896216] clocksource: Switched to clocksource kvm-clock [ 1.025308] VFS: Disk quotas dquot_6.6.0 [ 1.026682] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.029076] *** VALIDATE ramfs *** [ 1.030162] *** VALIDATE hugetlbfs *** [ 1.032028] pnp: PnP ACPI init [ 1.034322] pnp: PnP ACPI: found 6 devices [ 1.050770] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.053887] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.056054] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.058023] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.060235] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.062511] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.065372] NET: Registered protocol family 2 [ 1.067494] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.072160] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.074928] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.080190] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.083585] TCP: Hash tables configured (established 65536 bind 65536) [ 1.086535] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.089579] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.092496] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.095515] NET: Registered protocol family 1 [ 1.098110] RPC: Registered named UNIX socket transport module. [ 1.100281] RPC: Registered udp transport module. [ 1.101698] RPC: Registered tcp transport module. [ 1.103528] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.105713] NET: Registered protocol family 44 [ 1.107480] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.109411] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.111838] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.114080] PCI: CLS 0 bytes, default 64 [ 1.115571] Unpacking initramfs... [ 2.936369] debug: unmapping init [mem 0xffff907bbcc54000-0xffff907bbffbffff] [ 2.941484] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.944606] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.948583] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.534610] Initialise system trusted keyrings [ 3.536429] Key type blacklist registered [ 3.538363] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.548166] zbud: loaded [ 3.551355] *** VALIDATE nfs *** [ 3.552619] *** VALIDATE nfs4 *** [ 3.554397] pstore: using deflate compression [ 3.558428] Platform Keyring initialized [ 3.682692] NET: Registered protocol family 38 [ 3.686457] Key type asymmetric registered [ 3.690608] Asymmetric key parser 'x509' registered [ 3.692800] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.696628] io scheduler mq-deadline registered [ 3.699119] io scheduler kyber registered [ 3.700987] io scheduler bfq registered [ 3.702640] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.706478] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.710206] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.713739] ACPI: Power Button [PWRF] [ 3.722368] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.730975] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.748523] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.758283] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.781351] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.812754] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.839980] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.845387] Non-volatile memory driver v1.3 [ 3.847436] Linux agpgart interface v0.103 [ 3.881199] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.885367] vda: detected capacity change from 0 to 74649600 [ 3.928254] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.933109] vdb: detected capacity change from 0 to 1073741824 [ 3.954024] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.956834] vdc: detected capacity change from 0 to 2621440000 [ 3.975108] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.978573] vdd: detected capacity change from 0 to 2621440000 [ 3.999064] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.002213] vde: detected capacity change from 0 to 4294967296 [ 4.030441] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.034197] vdf: detected capacity change from 0 to 4294967296 [ 4.045131] libphy: Fixed MDIO Bus: probed [ 4.053762] usbcore: registered new interface driver usbserial_generic [ 4.057198] usbserial: USB Serial support registered for generic [ 4.059913] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.065524] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.067677] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.070279] mousedev: PS/2 mouse device common for all mice [ 4.073351] rtc_cmos 00:05: RTC can wake from S4 [ 4.076290] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.080909] rtc_cmos 00:05: registered as rtc0 [ 4.080972] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.081046] intel_pstate: CPU model not supported [ 4.086418] hid: raw HID events driver (C) Jiri Kosina [ 4.097895] usbcore: registered new interface driver usbhid [ 4.099944] usbhid: USB HID core driver [ 4.101752] drop_monitor: Initializing network drop monitor service [ 4.105287] Initializing XFRM netlink socket [ 4.106218] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.109446] NET: Registered protocol family 10 [ 4.117621] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.125117] Segment Routing with IPv6 [ 4.126567] NET: Registered protocol family 17 [ 4.128977] mpls_gso: MPLS GSO support [ 4.135517] RAS: Correctable Errors collector initialized. [ 4.137815] AVX version of gcm_enc/dec engaged. [ 4.139554] AES CTR mode by8 optimization enabled [ 4.250514] sched_clock: Marking stable (4250428814, 0)->(5118103524, -867674710) [ 4.254522] registered taskstats version 1 [ 4.256426] Loading compiled-in X.509 certificates [ 4.258415] zswap: loaded using pool lzo/zbud [ 4.293935] Key type big_key registered [ 4.310801] Key type encrypted registered [ 4.314923] ima: No TPM chip found, activating TPM-bypass! [ 4.317222] ima: Allocated hash algorithm: sha1 [ 4.319208] ima: No architecture policies found [ 4.321217] evm: Initialising EVM extended attributes: [ 4.323306] evm: security.selinux [ 4.324524] evm: security.ima [ 4.325650] evm: security.capability [ 4.327169] evm: HMAC attrs: 0x1 [ 4.329818] rtc_cmos 00:05: setting system clock to 2026-07-15 05:20:35 UTC (1784092835) [ 4.336970] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 4.340369] debug: unmapping init [mem 0xffffffffb3f82000-0xffffffffb4258fff] [ 4.349103] Write protecting the kernel read-only data: 28672k [ 4.353335] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 4.356296] debug: unmapping init [mem 0xffffffffb2f14000-0xffffffffb2ffffff] [ 4.403319] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.411656] systemd[1]: Detected virtualization kvm. [ 4.413732] systemd[1]: Detected architecture x86-64. [ 4.415587] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.443861] systemd[1]: No hostname configured. [ 4.445756] systemd[1]: Set hostname to . [ 4.449588] random: systemd: uninitialized urandom read (16 bytes read) [ 4.452808] systemd[1]: Initializing machine ID from random generator. [ 4.574068] random: ln: uninitialized urandom read (6 bytes read) [ 4.737791] random: systemd: uninitialized urandom read (16 bytes read) [ 4.741664] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.755796] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.761579] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.249468] device-mapper: uevent: version 1.0.3 [ 6.252498] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 7.553825] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.103653] virtio_net virtio0 ens2: renamed from eth0 [ 8.120282] scsi host0: ata_piix [ 8.343089] scsi host1: ata_piix [ 8.346414] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.356266] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.839171] random: crng init done [ 13.842535] random: 7 urandom warning(s) missed due to ratelimiting [ 16.079222] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 17.252481] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.819551] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.527791] SELinux: Disabled at runtime. [ 20.634788] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.647682] systemd[1]: Detected virtualization kvm. [ 20.649429] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.930273] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.934399] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.940224] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.950903] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.957596] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.974140] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.011416] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 22.138117] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.253428] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.150665] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.257487] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.671843] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.813214] EDAC sbridge: Ver: 1.1.2 [ 29.044362] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 29.778864] NFS: Registering the id_resolver key type [ 29.796244] Key type id_resolver registered [ 29.804805] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg445-server login: [ 74.553934] libcfs: loading out-of-tree module taints kernel. [ 74.587923] Key type ._llcrypt registered [ 74.589808] Key type .llcrypt registered [ 74.639387] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 85.116402] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 85.876656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 85.883995] alg: No test for adler32 (adler32-zlib) [ 87.034477] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 87.492334] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 89.145782] Key type lgssc registered [ 90.087766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.502104] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 121.910678] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 129.067556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 129.088208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.205254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 130.226206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 130.274565] Lustre: lustre-MDT0000: new disk, initializing [ 130.336384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.350615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 132.511366] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.539384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.593484] Lustre: 6488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 140.613394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 140.616236] Lustre: Skipped 1 previous similar message [ 140.663393] Lustre: lustre-MDT0001: new disk, initializing [ 140.706529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 140.725032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 140.733554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 142.923144] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 146.068757] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 151.198050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 151.332944] Lustre: lustre-OST0000: new disk, initializing [ 151.337080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 151.340830] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 151.385103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.090921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 153.097879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 153.159080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 154.107740] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.286160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 161.371845] Lustre: lustre-OST0001: new disk, initializing [ 161.374636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 161.378790] Lustre: 9496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 161.413090] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.743990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 162.750359] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 162.801048] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 164.855421] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.010691] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 178.684739] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 181.986818] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing check_logdir /tmp/testlogs/ [ 184.340514] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing yml_node [ 186.606905] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 187.882189] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 189.284159] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 190.174930] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 15 01:23:40 EDT 2026 [ 199.053812] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 199.852830] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 213.984412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 213.986465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 213.992970] Lustre: Skipped 3 previous similar messages [ 213.995608] Lustre: Skipped 2 previous similar messages [ 218.690495] Lustre: server umount lustre-MDT0000 complete [ 220.598055] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093051 with bad export cookie 1890343592215902001 [ 220.599565] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 220.604027] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 224.227303] LustreError: 8430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 224.227440] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 224.230443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 224.230453] Lustre: Skipped 1 previous similar message [ 224.244890] LustreError: 8430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 224.262868] Lustre: Skipped 1 previous similar message [ 225.248428] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 225.250428] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 225.259277] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 225.262614] Lustre: Skipped 2 previous similar messages [ 226.890990] Lustre: server umount lustre-MDT0001 complete [ 229.399770] Lustre: server umount lustre-OST0000 complete [ 231.614185] Lustre: server umount lustre-OST0001 complete [ 235.361171] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 240.111502] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 262.341970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 262.485144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 262.504355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 262.560819] Lustre: lustre-MDT0000: new disk, initializing [ 262.626782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.636860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 264.879832] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 271.247213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 271.308389] Lustre: 15040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 271.351948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 271.355480] Lustre: Skipped 1 previous similar message [ 271.410636] Lustre: lustre-MDT0001: new disk, initializing [ 271.487274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.523740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 271.530752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 273.875796] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 279.174363] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 281.847456] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 285.319730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.453039] Lustre: 17009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 287.196518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 287.257109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 288.495108] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 294.064902] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 295.994876] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 296.093484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 297.854521] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 297.952138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 302.563728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 302.574378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.583372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 302.588056] Lustre: Skipped 1 previous similar message [ 307.682382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 307.688928] Lustre: Skipped 1 previous similar message [ 313.311254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 313.406044] Lustre: server umount lustre-OST0000 complete [ 317.410689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 317.416325] LustreError: Skipped 1 previous similar message [ 317.419706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.429556] Lustre: Skipped 1 previous similar message [ 317.435092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.438812] Lustre: Skipped 2 previous similar messages [ 321.206990] Lustre: server umount lustre-MDT0000 complete [ 322.955530] LustreError: 15032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093154 with bad export cookie 1890343592215903534 [ 322.958738] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.960689] LustreError: 15032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 323.111958] Lustre: server umount lustre-MDT0001 complete [ 325.065324] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:25:55 (1784093155) [ 326.157750] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 326.916928] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:25:57 (1784093157) [ 330.399035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 330.674173] LustreError: 19415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.718045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 330.721555] Lustre: Skipped 1 previous similar message [ 332.832629] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 335.841044] LustreError: 19416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.960120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.209173] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 342.860127] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 345.691855] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 349.555263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 352.557131] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 357.040247] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 367.072898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 367.077699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.089589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 367.096711] Lustre: Skipped 2 previous similar messages [ 367.101262] Lustre: Skipped 1 previous similar message [ 371.567445] Lustre: server umount lustre-OST0000 complete [ 377.313712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.322894] Lustre: Skipped 1 previous similar message [ 379.770953] Lustre: server umount lustre-MDT0000 complete [ 381.515525] LustreError: 19397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093212 with bad export cookie 1890343592215904416 [ 381.517421] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.520301] LustreError: 19397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 381.656604] Lustre: server umount lustre-MDT0001 complete [ 387.314093] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 388.927815] Key type lgssc unregistered [ 389.099362] LNet: 22947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.105957] LNetError: 22947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.119676] LNet: Removed LNI 192.168.204.145@tcp [ 389.652191] Key type .llcrypt unregistered [ 389.653995] Key type ._llcrypt unregistered [ 402.952899] Key type ._llcrypt registered [ 402.956560] Key type .llcrypt registered [ 403.013366] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 403.443195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 403.472178] alg: No test for adler32 (adler32-zlib) [ 404.377280] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 404.506471] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 406.111187] Key type lgssc registered [ 406.692165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 411.066223] loop: module loaded [ 411.095103] loop0: detected capacity change from 0 to 204800000 [ 411.182781] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 411.188972] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 411.343570] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 414.542484] loop0: detected capacity change from 0 to 204800000 [ 414.672063] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 414.680484] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 414.689850] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 417.821729] loop0: detected capacity change from 0 to 204800000 [ 420.868895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 420.882790] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.040191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.086767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.136994] Lustre: lustre-MDT0000: new disk, initializing [ 422.199214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.215320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 424.890164] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 430.638019] loop1: detected capacity change from 0 to 204800000 [ 434.084468] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.282857] Lustre: lustre-OST07c6: new disk, initializing [ 434.286380] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 434.289960] Lustre: Skipped 1 previous similar message [ 434.293838] Lustre: 26105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.342677] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 436.779258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 436.789411] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 436.820409] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 438.210312] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 442.311850] Lustre: 25093:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 442.322047] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 442.325322] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 442.329465] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 442.333828] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 442.338090] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 459.521504] Lustre: 25095:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 459.526805] Lustre: 25095:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 459.531745] Lustre: 25095:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 459.539241] Lustre: 25095:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 459.545752] Lustre: 25095:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 459.551651] Lustre: 25095:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 463.610356] Lustre: 25093:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 463.616802] Lustre: 25093:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 463.621276] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 463.624942] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 463.629839] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 463.634805] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 463.639532] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 463.644759] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 463.649262] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 463.654087] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 463.657931] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 463.662572] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 466.763331] Lustre: 25095:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 466.768788] Lustre: 25095:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 466.772845] Lustre: 25095:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 466.777475] Lustre: 25095:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 466.781305] Lustre: 25095:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 466.784608] Lustre: 25095:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 466.789029] Lustre: 25095:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 466.793913] Lustre: 25095:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 466.798330] Lustre: 25095:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 466.802519] Lustre: 25095:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 466.805938] Lustre: 25095:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 466.809818] Lustre: 25095:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 473.845800] Lustre: 25095:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 473.856840] Lustre: 25095:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 473.864799] Lustre: 25095:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 473.873448] Lustre: 25095:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 473.880532] Lustre: 25095:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 473.884477] Lustre: 25095:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 473.890939] Lustre: 25095:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 473.896424] Lustre: 25095:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 473.903857] Lustre: 25095:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 473.909443] Lustre: 25095:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 473.914926] Lustre: 25095:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 473.918877] Lustre: 25095:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 483.979334] Lustre: 25093:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 483.984793] Lustre: 25093:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 483.988992] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 483.993678] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 483.998180] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 484.003172] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 484.008200] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 484.012189] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 484.016705] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 484.021612] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 484.025726] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 484.030145] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 528.864517] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.874411] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 532.065614] Lustre: server umount lustre-OST07c6 complete [ 536.347692] Lustre: server umount lustre-MDT0000 complete [ 543.306806] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 544.561062] Key type lgssc unregistered [ 544.720722] LNet: 28665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 544.727732] LNetError: 28665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 544.738453] LNet: Removed LNI 192.168.204.145@tcp [ 545.135870] Key type .llcrypt unregistered [ 545.137318] Key type ._llcrypt unregistered [ 554.482610] Key type ._llcrypt registered [ 554.484372] Key type .llcrypt registered [ 554.533479] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 554.926032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 554.967271] alg: No test for adler32 (adler32-zlib) [ 555.866780] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 555.985807] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 557.583536] Key type lgssc registered [ 558.130273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 562.036454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 562.044840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.244747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 565.073503] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 568.578317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.739862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 570.417365] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 573.566748] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 576.194453] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 579.322245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 579.489105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 582.195542] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 585.276726] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 588.711581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 604.132241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.133665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 604.139971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 604.150510] Lustre: Skipped 1 previous similar message [ 608.425663] Lustre: server umount lustre-OST0000 complete [ 614.369750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.371573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 614.380945] Lustre: Skipped 2 previous similar messages [ 616.250839] Lustre: server umount lustre-MDT0000 complete [ 617.994577] LustreError: 32713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093449 with bad export cookie 10362975287202104615 [ 617.997149] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.001906] LustreError: 32713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 618.129773] Lustre: server umount lustre-MDT0001 complete [ 622.603234] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 623.898914] Key type lgssc unregistered [ 624.045555] LNet: 33664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.049882] LNetError: 33664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.059424] LNet: Removed LNI 192.168.204.145@tcp [ 624.468233] Key type .llcrypt unregistered [ 624.469879] Key type ._llcrypt unregistered [ 633.086350] Key type ._llcrypt registered [ 633.087870] Key type .llcrypt registered [ 633.137449] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 633.606376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.626191] alg: No test for adler32 (adler32-zlib) [ 634.528192] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 634.635286] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 636.239293] Key type lgssc registered [ 636.726195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 640.178370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 640.187799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.432340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 643.108816] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 647.133951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.311354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 649.136668] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 652.189461] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 654.565088] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 657.442213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 657.606272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.072370] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 662.982567] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 666.087546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 673.248469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 673.252471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.259621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 676.321720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 676.323877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 676.324165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.337148] Lustre: Skipped 1 previous similar message [ 678.497819] Lustre: server umount lustre-OST0000 complete [ 681.440794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.441558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 681.445890] Lustre: Skipped 1 previous similar message [ 681.450687] Lustre: Skipped 1 previous similar message [ 686.172174] Lustre: server umount lustre-MDT0000 complete [ 686.560963] LustreError: 34911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.566516] LustreError: 34911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 687.687231] LustreError: 34892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093518 with bad export cookie 565273206070227993 [ 687.690427] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 687.693037] LustreError: 34892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 687.803539] Lustre: server umount lustre-MDT0001 complete [ 691.955840] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 693.099089] Key type lgssc unregistered [ 693.234530] LNet: 38442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.238098] LNetError: 38442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.246412] LNet: Removed LNI 192.168.204.145@tcp [ 693.573418] Key type .llcrypt unregistered [ 693.574913] Key type ._llcrypt unregistered [ 703.595830] Key type ._llcrypt registered [ 703.597506] Key type .llcrypt registered [ 703.642777] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 704.102833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 704.110317] alg: No test for adler32 (adler32-zlib) [ 704.974969] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 705.080977] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 706.671400] Key type lgssc registered [ 707.152221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 710.271515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.281470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.485762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.051465] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 716.684679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.875958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 718.588238] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 721.555071] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 723.762922] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 726.736534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 726.884954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.506261] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 732.604800] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 736.104258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 741.345729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.349728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.358535] Lustre: Skipped 2 previous similar messages [ 741.361583] Lustre: Skipped 1 previous similar message [ 742.453819] Lustre: server umount lustre-MDT0000 complete [ 743.052769] LustreError: 39828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 743.062487] LustreError: 39828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 744.094952] LustreError: 39815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093575 with bad export cookie 8152835350707248933 [ 744.097746] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.100754] LustreError: 39815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 746.311920] Lustre: server umount lustre-MDT0001 complete [ 757.449818] LustreError: 39816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.45@tcp arrived at 1784093588 with bad export cookie 8152835350707249556 [ 757.456296] LustreError: 39816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 774.813186] Lustre: server umount lustre-OST0000 complete [ 781.294873] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 782.491644] Key type lgssc unregistered [ 782.643636] LNet: 43688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.647762] LNetError: 43688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.656480] LNet: Removed LNI 192.168.204.145@tcp [ 783.031949] Key type .llcrypt unregistered [ 783.034138] Key type ._llcrypt unregistered [ 793.418837] Key type ._llcrypt registered [ 793.420665] Key type .llcrypt registered [ 793.471063] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 793.828610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 793.868093] alg: No test for adler32 (adler32-zlib) [ 794.752095] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 794.854463] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 796.447149] Key type lgssc registered [ 796.913343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.252367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 800.263423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.478420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 803.158813] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 806.869237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.058759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.815356] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 811.753428] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 814.032400] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 816.880555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 817.036865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.569815] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 822.855378] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 826.278881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 831.457239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 831.459405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.459919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 831.473621] Lustre: Skipped 1 previous similar message [ 833.681729] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 833.686512] Lustre: Skipped 1 previous similar message [ 836.579180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 836.584907] Lustre: Skipped 1 previous similar message [ 838.783743] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 843.903996] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 843.909702] Lustre: Skipped 2 previous similar messages [ 845.279165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 845.343395] Lustre: server umount lustre-OST0000 complete [ 858.540540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.541286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 858.545298] Lustre: Skipped 1 previous similar message [ 858.551039] Lustre: Skipped 1 previous similar message [ 864.810966] Lustre: server umount lustre-MDT0000 complete [ 866.387317] LustreError: 45065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093697 with bad export cookie 15960929783127796985 [ 866.391709] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 866.393966] LustreError: 45065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 872.575700] Lustre: server umount lustre-MDT0001 complete [ 876.871438] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 878.061647] Key type lgssc unregistered [ 878.194644] LNet: 48790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.198477] LNetError: 48790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.209418] LNet: Removed LNI 192.168.204.145@tcp [ 878.573482] Key type .llcrypt unregistered [ 878.575234] Key type ._llcrypt unregistered [ 888.950886] Key type ._llcrypt registered [ 888.952625] Key type .llcrypt registered [ 889.014293] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 889.388767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 889.456598] alg: No test for adler32 (adler32-zlib) [ 890.342722] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 890.459750] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 892.055293] Key type lgssc registered [ 892.503899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 895.717319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 895.728087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.947743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.698741] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 902.445302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.631208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 904.375464] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 907.494452] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.856567] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 912.749135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 912.906527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.411381] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 918.507644] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 922.152977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 930.912990] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 932.460123] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 943.313778] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-15 1h36m13s [ 943.943141] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 945.287735] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 946.301118] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 947.681934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.684114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.688074] Lustre: Skipped 2 previous similar messages [ 947.690811] Lustre: Skipped 1 previous similar message [ 951.016419] Lustre: server umount lustre-MDT0000 complete [ 952.447271] LustreError: 50179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.456756] LustreError: 50179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 952.540528] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 954.849690] LustreError: 50184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.859582] LustreError: 50184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 955.416214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.512439] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 955.622921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.652211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 956.648083] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 957.235672] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 958.573675] Lustre: DEBUG MARKER: Wait for df (36574) ... [ 959.161122] Lustre: DEBUG MARKER: done [ 960.992198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 960.996769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.002922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.005231] Lustre: Skipped 4 previous similar messages [ 962.018806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.324410] Lustre: server umount lustre-OST0000 complete [ 965.715761] Lustre: server umount lustre-MDT0000 complete [ 966.890353] LustreError: 50166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093798 with bad export cookie 16036060829360203529 [ 966.894643] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 966.894820] LustreError: 50166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 973.040324] Lustre: server umount lustre-MDT0001 complete [ 976.389186] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 977.353163] Key type lgssc unregistered [ 977.473364] LNet: 55119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.475910] LNetError: 55119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.485297] LNet: Removed LNI 192.168.204.145@tcp [ 977.781683] Key type .llcrypt unregistered [ 977.782820] Key type ._llcrypt unregistered [ 992.093688] Key type ._llcrypt registered [ 992.095018] Key type .llcrypt registered [ 992.131733] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 992.552465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.563210] alg: No test for adler32 (adler32-zlib) [ 993.421203] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 993.510858] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 995.087155] Key type lgssc registered [ 995.470874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 998.162612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 998.168058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.257641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 999.265970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 999.325329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.672295] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1003.529782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.568165] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1003.580824] Lustre: 57304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1003.689139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.124270] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1007.889531] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1010.009396] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1012.331067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.400641] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1012.468679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.490475] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1015.974554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1017.075494] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1020.493358] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-15 1h37m30s [ 1021.019808] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1023.402561] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1024.192700] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1024.221422] LustreError: 58574:0:(mdt_handler.c:2349:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1044.618243] Lustre: lustre-MDT0000: Client 9fe1f042-6f5f-4220-8e69-1a181d63cb54 (at 192.168.204.45@tcp) reconnecting [ 1064.263112] LustreError: 58574:0:(mdt_handler.c:2349:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1065.037467] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1065.843937] Lustre: DEBUG MARKER: done [ 1067.489801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1067.492624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.498208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.500805] Lustre: Skipped 1 previous similar message [ 1072.612701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1072.614826] Lustre: Skipped 1 previous similar message [ 1073.416624] Lustre: server umount lustre-OST0000 complete [ 1075.679864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1075.683457] LustreError: Skipped 1 previous similar message [ 1075.685598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.691599] Lustre: Skipped 1 previous similar message [ 1075.694500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.664893] Lustre: server umount lustre-MDT0000 complete [ 1081.928585] LustreError: 58045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093913 with bad export cookie 10325658023385676558 [ 1081.932452] LustreError: 58045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1081.932743] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1082.055119] Lustre: server umount lustre-MDT0001 complete [ 1085.602235] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1086.655069] Key type lgssc unregistered [ 1086.788435] LNet: 61623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.792180] LNetError: 61623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.799307] LNet: Removed LNI 192.168.204.145@tcp [ 1087.110671] Key type .llcrypt unregistered [ 1087.111950] Key type ._llcrypt unregistered [ 1102.296184] Key type ._llcrypt registered [ 1102.297738] Key type .llcrypt registered [ 1102.335765] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1102.680902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.744478] alg: No test for adler32 (adler32-zlib) [ 1103.613343] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 1103.710866] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1105.287144] Key type lgssc registered [ 1105.651930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1106.754683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1106.761178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.846362] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1107.854607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1107.914459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.261437] Lustre: Failing over lustre-MDT0000 [ 1108.364048] Lustre: server umount lustre-MDT0000 complete [ 1110.510711] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:39:00 (1784093940) [ 1112.707965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.843315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.166365] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1116.894165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.928886] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1116.939083] Lustre: 64396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1117.031404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1118.393589] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1120.669417] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1122.336699] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1124.596817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1124.670734] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1124.742560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.715786] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1127.823775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1129.018682] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1131.993720] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1137.631845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1137.634237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.639912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.144709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.146909] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.151301] Lustre: Skipped 1 previous similar message [ 1139.610233] Lustre: server umount lustre-MDT0000 complete [ 1140.750029] LustreError: 64388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784093971 with bad export cookie 10890771511125977200 [ 1140.753265] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.754153] LustreError: 64388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1140.851534] Lustre: server umount lustre-MDT0001 complete [ 1141.913727] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1143.380048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1146.105471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.431843] LustreError: 63319:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a6fa215f x1870758080968832/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1150.577435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.604634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1151.833688] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1154.537060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.637653] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.998886] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1158.546342] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1158.757151] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1159.775076] Lustre: 63320:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784093974/real 1784093974] req@0000000071745dc4 x1870758080968448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784093990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1160.418095] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1163.103122] Lustre: 63321:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784093976/real 1784093976] req@00000000f7545146 x1870758080968576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784093992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1164.127142] Lustre: 63322:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784093979/real 1784093979] req@0000000010ed2853 x1870758080968704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784093995 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1164.256630] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.258092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.261776] Lustre: Skipped 2 previous similar messages [ 1164.264142] Lustre: Skipped 3 previous similar messages [ 1166.303164] Lustre: 63323:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784093981/real 1784093981] req@0000000079d2838a x1870758080968960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784093997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1168.211399] Lustre: server umount lustre-MDT0000 complete [ 1169.376851] LustreError: 68158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.382628] LustreError: 68158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1169.415273] LustreError: 64388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094000 with bad export cookie 10890771511125979468 [ 1169.418046] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.418706] LustreError: 64388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1169.515661] Lustre: server umount lustre-MDT0001 complete [ 1171.614153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1174.831875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.616239] LustreError: 63319:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004932ea99 x1870758080988032/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1179.750458] LustreError: 68888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.776636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.779461] Lustre: Skipped 1 previous similar message [ 1179.802387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1181.188275] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1184.028697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.128575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1185.497322] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1187.977053] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1189.349381] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1189.351967] Lustre: Skipped 1 previous similar message [ 1189.830419] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1190.367117] Lustre: 63323:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094005/real 1784094005] req@0000000027ebff46 x1870758080987776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784094021 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1194.467234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1194.471057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1194.477590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1194.480428] Lustre: Skipped 2 previous similar messages [ 1197.657301] Lustre: server umount lustre-MDT0000 complete [ 1198.909287] LustreError: 65130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094030 with bad export cookie 10890771511125980847 [ 1198.911877] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.914491] LustreError: 65130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1199.013989] Lustre: server umount lustre-MDT0001 complete [ 1201.081920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1202.158608] Lustre: DEBUG MARKER: files compared the same [ 1212.059662] Lustre: server umount lustre-OST0000 complete [ 1217.253343] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1218.223591] Key type lgssc unregistered [ 1218.347294] LNet: 74105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.349589] LNetError: 74105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.356251] LNet: Removed LNI 192.168.204.145@tcp [ 1218.634195] Key type .llcrypt unregistered [ 1218.635484] Key type ._llcrypt unregistered [ 1226.921092] Key type ._llcrypt registered [ 1226.922205] Key type .llcrypt registered [ 1226.958375] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1227.261374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1227.315084] alg: No test for adler32 (adler32-zlib) [ 1228.156922] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 1228.237859] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1229.815156] Key type lgssc registered [ 1230.140081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1232.520939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1232.526389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1233.659104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.859335] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 1237.357410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.445909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1238.612187] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 1240.704512] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1242.302109] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1244.373406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1244.458416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.151459] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 1248.316460] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1258.786160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1268.191886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1268.198738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.203893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.216157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.218122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1269.225166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.418540] Lustre: server umount lustre-OST0000 complete [ 1278.432376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1278.446095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.471604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1278.480439] Lustre: Skipped 1 previous similar message [ 1280.886339] Lustre: server umount lustre-MDT0000 complete [ 1284.379148] LustreError: 75479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094115 with bad export cookie 17845659221965733918 [ 1284.390535] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.390763] LustreError: 75479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1284.678962] Lustre: server umount lustre-MDT0001 complete [ 1293.624909] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1296.155709] Key type lgssc unregistered [ 1296.366639] LNet: 79048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.372721] LNetError: 79048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.396915] LNet: Removed LNI 192.168.204.145@tcp [ 1297.019178] Key type .llcrypt unregistered [ 1297.021252] Key type ._llcrypt unregistered [ 1313.202531] Key type ._llcrypt registered [ 1313.204694] Key type .llcrypt registered [ 1313.243981] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1313.723187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1313.730904] alg: No test for adler32 (adler32-zlib) [ 1314.645344] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 1314.777784] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1316.384486] Key type lgssc registered [ 1316.965178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1321.442993] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1321.458863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.991990] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1342.111396] LustreError: 80416:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 1368.735172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1372.040524] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1375.934300] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1379.757889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.830496] Lustre: *** cfs_fail_loc=706, val=0*** [ 1380.035989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.078499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1382.131960] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1386.671794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.860696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.918314] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1392.590765] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1395.347819] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1397.217095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1397.222425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.230326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1402.341475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1402.344643] Lustre: Skipped 1 previous similar message [ 1402.407808] Lustre: server umount lustre-OST0000 complete [ 1407.458132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1407.458318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.461523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1407.461529] Lustre: Skipped 1 previous similar message [ 1407.476616] Lustre: Skipped 2 previous similar messages [ 1410.351278] Lustre: server umount lustre-MDT0000 complete [ 1412.106267] LustreError: 80423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094243 with bad export cookie 4509389248648288724 [ 1412.109895] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1412.114087] LustreError: 80423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1412.246937] Lustre: server umount lustre-MDT0001 complete [ 1417.137894] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1418.580758] Key type lgssc unregistered [ 1418.752666] LNet: 84028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.756625] LNetError: 84028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1418.765478] LNet: Removed LNI 192.168.204.145@tcp [ 1419.192910] Key type .llcrypt unregistered [ 1419.195067] Key type ._llcrypt unregistered [ 1431.179422] Key type ._llcrypt registered [ 1431.181104] Key type .llcrypt registered [ 1431.228319] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1431.753837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.781207] alg: No test for adler32 (adler32-zlib) [ 1432.677177] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 1432.787528] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1434.383335] Key type lgssc registered [ 1434.975850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1439.114816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1439.127641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.420364] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1446.508029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.814242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1448.815285] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1452.001612] LustreError: 86107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.013474] LustreError: 86107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1453.141443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.325708] LustreError: 86106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.336947] LustreError: 86106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1453.346699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.089320] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1456.417464] LustreError: 86110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.429576] LustreError: 86110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1460.245708] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1460.250834] Lustre: MGS: Client cdc53c28-7243-4983-bcb0-c37fce810fb9 (at 0@lo) reconnecting [ 1460.261522] LustreError: 86121:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1460.368795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.403911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1462.184909] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1471.969078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1471.972326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.982213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.065477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.066918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1476.072854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.741680] Lustre: server umount lustre-OST0000 complete [ 1481.185276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.186633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1481.187223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.187228] Lustre: Skipped 1 previous similar message [ 1481.192104] Lustre: Skipped 1 previous similar message [ 1485.902785] Lustre: server umount lustre-MDT0000 complete [ 1486.308387] LustreError: 86111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.314885] LustreError: 86111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1487.817200] Lustre: server umount lustre-MDT0001 complete [ 1489.569404] Lustre: server umount MGS complete [ 1495.460660] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:45:25 (1784094325) [ 1510.178097] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 1515.262855] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1531.528206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.600280] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1539.483316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.637461] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1539.674687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1539.728306] Lustre: lustre-MDT0001: new disk, initializing [ 1539.782783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1539.814509] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1539.820076] Lustre: 93266:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1539.828783] LustreError: 93266:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1542.106996] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1548.056215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1548.177502] Lustre: lustre-OST0000: new disk, initializing [ 1548.181910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1548.186181] Lustre: 94031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1548.225812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.032915] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1555.847146] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1558.562240] Lustre: 94895:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000052af50b1 x1870758425955456/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784094405 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1558.899996] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1558.907334] Lustre: MGS: Client ed9cdc80-c4a3-4730-9d55-44beced64171 (at 0@lo) reconnecting [ 1558.928555] Lustre: 95074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1558.940203] LustreError: 93252:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1558.944476] LustreError: 93252:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1558.957483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1558.997924] Lustre: lustre-MDT0000: new disk, initializing [ 1559.070910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.085599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1560.160066] Lustre: 95090:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f842859f x1870758425960960/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784094407 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1560.171411] Lustre: 95090:0:(client.c:1639:after_reply()) Skipped 1 previous similar message [ 1560.607579] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1561.057044] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1569.256927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1569.266510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1569.289090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1593.313580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1593.314710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.315535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1593.315540] Lustre: Skipped 1 previous similar message [ 1593.328355] Lustre: Skipped 1 previous similar message [ 1597.598528] Lustre: server umount lustre-OST0000 complete [ 1603.552564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.554805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1603.584267] Lustre: Skipped 1 previous similar message [ 1603.604458] Lustre: Skipped 2 previous similar messages [ 1605.990490] Lustre: server umount lustre-MDT0000 complete [ 1608.673433] LustreError: 93237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1608.711438] LustreError: 93237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1616.957562] Lustre: server umount lustre-MDT0001 complete [ 1620.735904] Lustre: server umount MGS complete [ 1631.725691] hrtimer: interrupt took 11524780 ns [ 1632.616531] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:47:42 (1784094462) [ 1652.912586] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1656.823565] Key type lgssc unregistered [ 1657.242807] LNet: 97597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.260523] LNetError: 97597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.283653] LNet: Removed LNI 192.168.204.145@tcp [ 1658.490135] Key type .llcrypt unregistered [ 1658.492404] Key type ._llcrypt unregistered [ 1678.630316] Key type ._llcrypt registered [ 1678.639098] Key type .llcrypt registered [ 1678.819985] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1680.154641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.166117] alg: No test for adler32 (adler32-zlib) [ 1681.287122] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 1681.526765] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1683.303247] Key type lgssc registered [ 1684.720940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1692.183196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1692.205644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.612952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.571695] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1702.880824] LustreError: 98820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.898417] LustreError: 98820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1704.736131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.048093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1708.839277] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1714.734624] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1719.223596] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1724.735869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.030211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.602269] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1734.851257] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1738.354789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 1745.888350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1745.895773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.914260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1748.967161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1748.967549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1748.969755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.985987] Lustre: Skipped 1 previous similar message [ 1751.193036] Lustre: server umount lustre-OST0000 complete [ 1756.130855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.136773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.146951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.200645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.204996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.472628] Lustre: server umount lustre-MDT0000 complete [ 1763.287394] LustreError: 98801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094594 with bad export cookie 17474022297769035460 [ 1763.293987] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.294144] LustreError: 98801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1763.523522] Lustre: server umount lustre-MDT0001 complete [ 1772.325959] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1774.607788] Key type lgssc unregistered [ 1774.869984] LNet: 102353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.878065] LNetError: 102353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.892622] LNet: Removed LNI 192.168.204.145@tcp [ 1775.660391] Key type .llcrypt unregistered [ 1775.664139] Key type ._llcrypt unregistered [ 1787.838578] Key type ._llcrypt registered [ 1787.840859] Key type .llcrypt registered [ 1787.926272] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1788.611766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.648061] alg: No test for adler32 (adler32-zlib) [ 1789.672964] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 1789.883438] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1791.567174] Key type lgssc registered [ 1792.528732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.316214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.323662] LustreError: 103420:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1797.370254] LustreError: 103430:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1797.376400] LustreError: 103420:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4954 [ 1797.381684] LustreError: 103420:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1797.396836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.409417] LustreError: 103420:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1798.830375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.966860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.274833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1808.655532] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1813.057340] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lsmod [ 1815.335655] LustreError: 104634:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1815.352412] LustreError: 104638:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1815.358526] LustreError: 104634:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 1815.366799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1815.370076] LustreError: 104634:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1815.370131] LustreError: 104634:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1815.529839] LustreError: 104650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1815.543702] LustreError: 104650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1815.680831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.645519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1820.655302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.673374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.156192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.172855] Lustre: Skipped 1 previous similar message [ 1821.177530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.188500] Lustre: Skipped 2 previous similar messages [ 1824.351672] Lustre: server umount lustre-MDT0000 complete [ 1826.275751] LustreError: 104473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.291553] LustreError: 104473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1827.156816] LustreError: 103442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094658 with bad export cookie 7928726021981982699 [ 1827.164439] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.169139] LustreError: 103442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1827.313263] LustreError: 103898:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007f36a53e x1870758801328896/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1827.441496] Lustre: server umount lustre-MDT0001 complete [ 1840.822891] Lustre: server umount lustre-OST0000 complete [ 1847.342594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.758339] LustreError: 105699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.833275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.392656] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1852.899461] LustreError: 105700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.993281] LustreError: 105699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.700124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.002719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1860.993254] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1866.242388] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1870.072710] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1875.604729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.968977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.001938] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1886.179392] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1892.401216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 1902.564043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.572466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.574714] Lustre: Skipped 1 previous similar message [ 1902.575940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1902.581825] Lustre: Skipped 1 previous similar message [ 1907.415712] Lustre: server umount lustre-OST0000 complete [ 1912.803025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.813547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1912.821110] Lustre: Skipped 1 previous similar message [ 1916.547798] Lustre: server umount lustre-MDT0000 complete [ 1917.920864] LustreError: 105699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.943333] LustreError: 105699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1919.543246] LustreError: 105679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094750 with bad export cookie 7928726021981983399 [ 1919.545197] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.552702] LustreError: 105679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1919.826901] Lustre: server umount lustre-MDT0001 complete [ 1928.046700] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1930.259857] Key type lgssc unregistered [ 1930.516195] LNet: 109251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.525022] LNetError: 109251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.543792] LNet: Removed LNI 192.168.204.145@tcp [ 1931.272072] Key type .llcrypt unregistered [ 1931.275258] Key type ._llcrypt unregistered [ 1949.587410] Key type ._llcrypt registered [ 1949.589484] Key type .llcrypt registered [ 1949.668909] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1950.460505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1950.479589] alg: No test for adler32 (adler32-zlib) [ 1951.524573] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 1951.686436] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1953.311167] Key type lgssc registered [ 1954.238377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1961.048697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1961.066877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.505607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1966.759184] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1972.709221] LustreError: 110647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.745915] LustreError: 110647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1974.893826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.399875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.868692] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1986.021992] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1990.446574] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1996.124397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1996.424179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.108508] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2006.964870] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2007.727727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 2012.115071] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2014.339191] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2018.277387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.285030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.289207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2018.296831] Lustre: Skipped 1 previous similar message [ 2018.307877] Lustre: Skipped 1 previous similar message [ 2019.616498] Lustre: server umount lustre-OST0000 complete [ 2023.393553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.401748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.416027] Lustre: Skipped 1 previous similar message [ 2023.429889] Lustre: Skipped 1 previous similar message [ 2028.522395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2028.532639] Lustre: Skipped 1 previous similar message [ 2028.694253] Lustre: server umount lustre-MDT0000 complete [ 2032.023612] LustreError: 110629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094863 with bad export cookie 16914529549117932506 [ 2032.030618] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2032.036315] LustreError: 110629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2032.297632] Lustre: server umount lustre-MDT0001 complete [ 2041.090527] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2043.490910] Key type lgssc unregistered [ 2043.769691] LNet: 114285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.785599] LNetError: 114285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.800797] LNet: Removed LNI 192.168.204.145@tcp [ 2044.516301] Key type .llcrypt unregistered [ 2044.520482] Key type ._llcrypt unregistered [ 2057.934910] Key type ._llcrypt registered [ 2057.937651] Key type .llcrypt registered [ 2058.037137] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2058.955149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.991185] alg: No test for adler32 (adler32-zlib) [ 2059.989415] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 2060.230900] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2061.879423] Key type lgssc registered [ 2062.737657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2067.797642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2067.815435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.148669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.004753] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2078.084286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.371483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.311427] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2086.568135] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2090.146830] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2094.033042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.260046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.271768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2097.988545] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2102.632390] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2110.944796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2110.961415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.982871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2114.017987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2114.019441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.031957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2115.842663] Lustre: server umount lustre-OST0000 complete [ 2119.145695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.147116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.155914] Lustre: Skipped 1 previous similar message [ 2119.162016] Lustre: Skipped 1 previous similar message [ 2124.256959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.263258] Lustre: Skipped 1 previous similar message [ 2124.549424] Lustre: server umount lustre-MDT0000 complete [ 2126.925263] LustreError: 115464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784094958 with bad export cookie 13712268270829667320 [ 2126.931118] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.933674] LustreError: 115464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2127.238653] Lustre: server umount lustre-MDT0001 complete [ 2135.091034] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2137.013269] Key type lgssc unregistered [ 2137.226172] LNet: 119014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.233626] LNetError: 119014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.247479] LNet: Removed LNI 192.168.204.145@tcp [ 2137.806968] Key type .llcrypt unregistered [ 2137.811225] Key type ._llcrypt unregistered [ 2154.283526] Key type ._llcrypt registered [ 2154.285679] Key type .llcrypt registered [ 2154.380627] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2154.919891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.956045] alg: No test for adler32 (adler32-zlib) [ 2155.937510] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 2156.143487] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2157.815506] Key type lgssc registered [ 2158.630416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2164.458981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2164.473704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.772086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.493860] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2174.398882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.718331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.448323] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2182.077864] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2185.760583] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2190.394344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2190.650130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.996259] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2200.387432] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2210.118615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 2216.495584] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2218.994465] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2218.997994] Lustre: Skipped 2 previous similar messages [ 2221.751355] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2221.754991] Lustre: Skipped 2 previous similar messages [ 2224.163795] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2224.178602] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2233.262394] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2233.269185] Lustre: Skipped 2 previous similar messages [ 2240.542472] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp [ 2244.533498] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.45/24@tcp in log lustre-MDT0000 [ 2244.540773] Lustre: Skipped 2 previous similar messages [ 2249.243891] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 2249.248837] Lustre: Skipped 1 previous similar message [ 2253.457601] Lustre: server umount lustre-OST0000 complete [ 2256.353242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.356807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2256.361446] Lustre: Skipped 1 previous similar message [ 2256.370263] Lustre: Skipped 1 previous similar message [ 2261.473806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2261.478115] Lustre: Skipped 1 previous similar message [ 2265.633991] Lustre: server umount lustre-MDT0000 complete [ 2266.595599] LustreError: 122661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.605684] LustreError: 122661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2267.962311] LustreError: 120391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784095099 with bad export cookie 11849252893330084517 [ 2267.965960] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2267.969829] LustreError: 120391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2278.479608] Lustre: server umount lustre-MDT0001 complete [ 2285.240210] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2286.942557] Key type lgssc unregistered [ 2287.138618] LNet: 125460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.149605] LNetError: 125460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.162705] LNet: Removed LNI 192.168.204.145@tcp [ 2287.692659] Key type .llcrypt unregistered [ 2287.695148] Key type ._llcrypt unregistered [ 2304.439584] Key type ._llcrypt registered [ 2304.442829] Key type .llcrypt registered [ 2304.499895] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2305.084655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2305.102479] alg: No test for adler32 (adler32-zlib) [ 2306.023211] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 2306.176189] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2307.807876] Key type lgssc registered [ 2308.526294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2313.065462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2313.079251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.332801] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2314.345260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.349627] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.45/24@tcp [ 2316.695640] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2321.912159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.160557] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2322.185040] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 2322.218777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2325.096835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2329.797363] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2333.587505] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2337.842248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2338.131471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.188344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 2341.839858] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2346.506731] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2354.656100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2354.663432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.678762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.728185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2357.734607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.735598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.829498] Lustre: server umount lustre-OST0000 complete [ 2362.850466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.865541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2362.871169] Lustre: Skipped 2 previous similar messages [ 2367.142330] Lustre: server umount lustre-MDT0000 complete [ 2367.969411] LustreError: 127000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.981766] LustreError: 127000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2369.234412] LustreError: 127740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784095200 with bad export cookie 4868858659559101164 [ 2369.241023] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2369.246387] LustreError: 127740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2369.450441] Lustre: server umount lustre-MDT0001 complete [ 2375.767268] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2377.528072] Key type lgssc unregistered [ 2377.707488] LNet: 130584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2377.712414] LNetError: 130584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.732743] LNet: Removed LNI 192.168.204.145@tcp [ 2378.295112] Key type .llcrypt unregistered [ 2378.301465] Key type ._llcrypt unregistered [ 2393.695881] Key type ._llcrypt registered [ 2393.698747] Key type .llcrypt registered [ 2393.766571] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 2403.389629] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2404.012112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2404.060425] alg: No test for adler32 (adler32-zlib) [ 2405.072578] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 2405.257587] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2406.895255] Key type lgssc registered [ 2407.727771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2428.046055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2428.062560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.201841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2429.230658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2429.298202] Lustre: lustre-MDT0000: new disk, initializing [ 2429.352265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.365470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2432.089870] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2439.611882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.705083] Lustre: 133959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2439.771386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2439.775447] Lustre: Skipped 1 previous similar message [ 2439.859797] Lustre: lustre-MDT0001: new disk, initializing [ 2439.925633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.949862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2439.957574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2442.874640] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2449.909723] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2453.714463] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2458.195470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2458.378909] Lustre: lustre-OST0000: new disk, initializing [ 2458.381865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2458.386293] Lustre: 135939:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2458.443523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.001612] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2466.845828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2466.859905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2466.944109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2468.177276] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2471.218355] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2471.357688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2474.126718] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2474.274708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2477.028972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2477.035700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.038492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.049598] Lustre: Skipped 1 previous similar message [ 2477.056079] Lustre: Skipped 1 previous similar message [ 2481.522160] Lustre: server umount lustre-OST0000 complete [ 2486.240438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2486.245783] LustreError: Skipped 1 previous similar message [ 2486.248658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.257824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.265274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.266279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.280230] Lustre: Skipped 1 previous similar message [ 2489.844598] Lustre: server umount lustre-MDT0000 complete [ 2491.748090] LustreError: 133952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784095322 with bad export cookie 450408366404497531 [ 2491.749112] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.754271] LustreError: 133952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2491.925894] Lustre: server umount lustre-MDT0001 complete [ 2496.715480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.066560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.381986] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2502.113031] LustreError: 138155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.124341] LustreError: 138155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2504.332590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.617280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2507.260822] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2511.604803] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2514.749796] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2519.031062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.230452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.715492] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2527.088404] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2529.707020] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2529.855207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2532.428952] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2532.582427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2539.143418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.251775] Lustre: lustre-OST0001: new disk, initializing [ 2539.254416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2539.261393] Lustre: 141523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2539.319827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2543.203832] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2545.212794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2545.230419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2545.265304] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2548.823232] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2548.974582] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2552.022322] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2552.187612] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2556.370261] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa01d60bd7800.ost_server_uuid 50 [ 2557.517727] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa01d60bd7800.ost_server_uuid in FULL state after 0 sec [ 2565.609315] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.619550] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2565.624884] Lustre: Skipped 1 previous similar message [ 2568.790908] Lustre: server umount lustre-OST0001 complete [ 2570.728960] LustreError: 140951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.747402] LustreError: 140951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2571.745572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2571.751181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.758247] Lustre: Skipped 1 previous similar message [ 2571.761973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.793602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2575.845498] LustreError: 139972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.856845] LustreError: 139972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2577.052991] Lustre: server umount lustre-OST0000 complete [ 2580.962101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.965414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.972163] Lustre: Skipped 2 previous similar messages [ 2580.981831] Lustre: Skipped 4 previous similar messages [ 2585.581788] Lustre: server umount lustre-MDT0000 complete [ 2586.084165] LustreError: 138149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.097339] LustreError: 138149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2588.021934] LustreError: 139994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784095419 with bad export cookie 450408366404498413 [ 2588.028301] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2588.032368] LustreError: 139994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2588.248355] Lustre: server umount lustre-MDT0001 complete [ 2599.610683] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2601.118934] Key type lgssc unregistered [ 2601.282236] LNet: 144519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2601.288079] LNetError: 144519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2601.298084] LNet: Removed LNI 192.168.204.145@tcp [ 2601.739167] Key type .llcrypt unregistered [ 2601.741269] Key type ._llcrypt unregistered [ 2623.924994] Key type ._llcrypt registered [ 2623.927699] Key type .llcrypt registered [ 2623.980535] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 2633.041244] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2633.724351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2633.761778] alg: No test for adler32 (adler32-zlib) [ 2634.682238] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 2634.820506] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2636.431222] Key type lgssc registered [ 2637.112301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2657.457213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2657.468229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.614409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2658.631234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2658.676720] Lustre: lustre-MDT0000: new disk, initializing [ 2658.721292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.734818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2661.338731] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2668.009211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.104661] Lustre: 148646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2668.148126] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2668.153397] Lustre: Skipped 1 previous similar message [ 2668.241833] Lustre: lustre-MDT0001: new disk, initializing [ 2668.313295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2668.341731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2668.355087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2670.653089] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2676.042096] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2679.029836] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2683.091044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.268211] Lustre: lustre-OST0000: new disk, initializing [ 2683.273801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2683.280489] Lustre: 150613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2683.342520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.054849] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2688.548773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2688.554279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2688.599241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2693.061292] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2696.742791] Lustre: Failing over lustre-OST0000 [ 2696.848840] Lustre: server umount lustre-OST0000 complete [ 2711.469449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.663743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.673878] Lustre: Skipped 1 previous similar message [ 2711.686933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.694727] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2714.414561] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2714.767685] Lustre: 146874:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784095529/real 1784095529] req@0000000088b6ce91 x1870759686336640/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784095545 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2715.615167] Lustre: 146873:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784095529/real 1784095529] req@000000003007eeb6 x1870759686336512/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784095545 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2715.789930] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2716.381206] Lustre: Failing over lustre-MDT0000 [ 2716.513755] Lustre: server umount lustre-MDT0000 complete [ 2717.153527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2717.162635] Lustre: Skipped 2 previous similar messages [ 2720.223159] Lustre: 146871:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784095535/real 1784095535] req@000000001268e07c x1870759686338432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784095551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2720.239363] Lustre: 146871:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2722.272550] LustreError: 148658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.281879] LustreError: 148658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2725.343451] Lustre: 146873:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784095540/real 1784095540] req@000000007f884a24 x1870759686339584/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784095556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2726.015478] LustreError: 148652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.394056] LustreError: 148657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.408286] LustreError: 148657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2731.134864] LustreError: 148652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.623321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.696981] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2731.705892] Lustre: 148649:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2731.836608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2731.878150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.914421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2734.415691] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2736.311399] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2737.127075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2737.138097] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2737.182694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2742.242700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2742.244678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.251470] LustreError: Skipped 1 previous similar message [ 2742.264184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.272181] Lustre: Skipped 1 previous similar message [ 2745.749865] Lustre: server umount lustre-OST0000 complete [ 2752.482990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.483662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2752.483916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2752.495309] Lustre: Skipped 2 previous similar messages [ 2754.176130] Lustre: server umount lustre-MDT0000 complete [ 2756.317499] LustreError: 148638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784095587 with bad export cookie 4726004225117297713 [ 2756.319855] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2756.324744] LustreError: 148638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2756.514874] Lustre: server umount lustre-MDT0001 complete [ 2762.828955] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2764.600277] Key type lgssc unregistered [ 2764.768588] LNet: 154262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2764.773785] LNetError: 154262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2764.783645] LNet: Removed LNI 192.168.204.145@tcp [ 2765.315231] Key type .llcrypt unregistered [ 2765.318095] Key type ._llcrypt unregistered [ 2790.073461] Key type ._llcrypt registered [ 2790.075186] Key type .llcrypt registered [ 2790.144725] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2790.757521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2790.770250] alg: No test for adler32 (adler32-zlib) [ 2791.688941] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 2791.853060] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2793.479209] Key type lgssc registered [ 2794.088272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2798.195134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2798.204393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.473276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.919783] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2807.414223] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2809.826489] LustreError: 155781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.832930] LustreError: 155781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2809.911082] LustreError: 155809:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2809.916099] LustreError: 155809:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2809.920642] LustreError: 155809:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 2810.088351] Lustre: server umount lustre-MDT0000 complete [ 2818.963306] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 2824.294368] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2841.417643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.525960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2841.541932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2841.585470] Lustre: lustre-MDT0000: new disk, initializing [ 2841.620575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.632054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2843.354409] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2849.023694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.094884] Lustre: 159106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2849.143597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2849.146224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2849.147903] Lustre: Skipped 1 previous similar message [ 2849.197140] Lustre: lustre-MDT0001: new disk, initializing [ 2849.245741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2849.271102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2849.279155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2851.319591] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2855.973807] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2858.503693] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2861.724393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2861.876755] Lustre: lustre-OST0000: new disk, initializing [ 2861.881549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2861.887262] Lustre: 161072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2861.938687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.687264] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2867.721062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2867.726223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2867.783576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2869.695257] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2871.936374] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2872.744595] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:08:23 (1784095703) [ 3000.799697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.800266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.808280] Lustre: Skipped 2 previous similar messages [ 3000.812029] Lustre: Skipped 1 previous similar message [ 3005.040510] Lustre: server umount lustre-MDT0000 complete [ 3005.919666] LustreError: 159113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.926617] LustreError: 159113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3006.806417] LustreError: 159099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784095837 with bad export cookie 10574711054262410636 [ 3006.807347] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.815366] LustreError: 159099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3011.042394] LustreError: 159112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.042493] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.066978] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3011.070219] Lustre: Skipped 1 previous similar message [ 3016.165335] LustreError: 159114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.172349] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3021.279387] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3021.385807] Lustre: server umount lustre-MDT0001 complete [ 3029.546780] Lustre: server umount lustre-OST0000 complete [ 3034.263722] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3039.439779] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3056.357681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.494561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3056.517956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3056.570745] Lustre: lustre-MDT0000: new disk, initializing [ 3056.613571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.624113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3058.874188] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3064.678780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.734386] Lustre: 165709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3064.780073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3064.783218] Lustre: Skipped 1 previous similar message [ 3064.835912] Lustre: lustre-MDT0001: new disk, initializing [ 3064.891656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3064.918494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3064.925820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3066.955208] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3072.101832] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3074.812251] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3078.126119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3078.262517] Lustre: lustre-OST0000: new disk, initializing [ 3078.265822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3078.269854] Lustre: 167678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3080.855096] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3084.305366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3084.311099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3084.348173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3085.557378] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3087.464579] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3087.555322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3089.480654] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3089.611531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3094.499642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.501238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3094.508496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3094.510774] LustreError: Skipped 1 previous similar message [ 3094.516988] Lustre: Skipped 2 previous similar messages [ 3096.556397] Lustre: server umount lustre-OST0000 complete [ 3099.616809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.624869] Lustre: Skipped 2 previous similar messages [ 3104.616489] Lustre: server umount lustre-MDT0000 complete [ 3104.736246] LustreError: 165722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.745134] LustreError: 165722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3106.297145] LustreError: 165701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784095937 with bad export cookie 10574711054262569473 [ 3106.301194] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.302799] LustreError: 165701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3106.435899] Lustre: server umount lustre-MDT0001 complete [ 3110.460996] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:12:20 (1784095940) [ 3117.616116] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3121.556214] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3137.195952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.304801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3137.325489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3137.369732] Lustre: lustre-MDT0000: new disk, initializing [ 3137.421378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.425283] Lustre: Skipped 1 previous similar message [ 3137.435931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3139.394512] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3144.775880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3144.834688] Lustre: 172428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3145.012489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3146.802986] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3151.525243] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3153.999615] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3156.996775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3157.139892] Lustre: 174403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3159.698299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3159.740407] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3159.754974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3164.220973] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3170.275172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3170.275647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.277677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3170.277684] Lustre: Skipped 2 previous similar messages [ 3170.292808] Lustre: Skipped 1 previous similar message [ 3172.695894] Lustre: server umount lustre-OST0000 complete [ 3180.443892] Lustre: server umount lustre-MDT0000 complete [ 3180.514302] LustreError: 172439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.521947] LustreError: 172439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3181.982615] LustreError: 175305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096013 with bad export cookie 10574711054262570341 [ 3181.985839] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3181.988371] LustreError: 175305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3182.125439] Lustre: server umount lustre-MDT0001 complete [ 3185.558067] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:13:35 (1784096015) [ 3192.549782] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3196.232755] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3210.255852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.362093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3210.374535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3210.377286] Lustre: Skipped 3 previous similar messages [ 3210.417360] Lustre: lustre-MDT0000: new disk, initializing [ 3210.419403] Lustre: Skipped 2 previous similar messages [ 3210.451650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.455243] Lustre: Skipped 2 previous similar messages [ 3210.464876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3210.470325] Lustre: Skipped 2 previous similar messages [ 3212.069392] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3216.933845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.993433] Lustre: 178717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3217.124218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3218.807467] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3223.362179] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3225.719983] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3228.606614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3228.743102] Lustre: 180694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3231.187182] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3235.612087] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3238.438809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3243.488622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3243.489975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3243.491438] LustreError: Skipped 1 previous similar message [ 3243.494964] Lustre: Skipped 2 previous similar messages [ 3243.495617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3243.501318] Lustre: Skipped 4 previous similar messages [ 3245.212222] Lustre: server umount lustre-OST0000 complete [ 3248.096000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3248.101414] LustreError: Skipped 1 previous similar message [ 3248.810623] Lustre: server umount lustre-MDT0000 complete [ 3250.291092] LustreError: 179518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096081 with bad export cookie 10574711054262571426 [ 3250.293830] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3250.296062] LustreError: 179518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3250.425225] Lustre: server umount lustre-MDT0001 complete [ 3254.684792] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3255.848973] Key type lgssc unregistered [ 3255.989336] LNet: 182547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3255.992777] LNetError: 182547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3256.001314] LNet: Removed LNI 192.168.204.145@tcp [ 3256.333945] Key type .llcrypt unregistered [ 3256.335446] Key type ._llcrypt unregistered [ 3266.396498] Key type ._llcrypt registered [ 3266.398282] Key type .llcrypt registered [ 3266.441189] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3266.988560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3267.000077] alg: No test for adler32 (adler32-zlib) [ 3267.880800] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 3267.992620] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3269.583209] Key type lgssc registered [ 3270.003569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3273.056826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3273.069454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.259116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.842985] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3279.051234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.211990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.694509] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3283.516676] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3285.794757] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3288.512966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.683588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.123409] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3294.094526] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3298.919447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3314.659833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3314.660323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.667328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.672707] Lustre: Skipped 1 previous similar message [ 3314.679844] Lustre: Skipped 1 previous similar message [ 3317.608611] Lustre: server umount lustre-OST0000 complete [ 3319.778337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.779886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3319.786622] Lustre: Skipped 1 previous similar message [ 3324.897448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.902812] Lustre: Skipped 2 previous similar messages [ 3325.347451] Lustre: server umount lustre-MDT0000 complete [ 3326.823320] LustreError: 184671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096157 with bad export cookie 3583957445263646986 [ 3326.826278] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.828747] LustreError: 184671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3326.958878] Lustre: server umount lustre-MDT0001 complete [ 3331.379449] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3332.548734] Key type lgssc unregistered [ 3332.705453] LNet: 187489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3332.709132] LNetError: 187489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3332.722471] LNet: Removed LNI 192.168.204.145@tcp [ 3333.094874] Key type .llcrypt unregistered [ 3333.097826] Key type ._llcrypt unregistered [ 3343.467619] Key type ._llcrypt registered [ 3343.469437] Key type .llcrypt registered [ 3343.519523] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3343.911502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3343.948080] alg: No test for adler32 (adler32-zlib) [ 3344.822221] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 3344.944565] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3346.543211] Key type lgssc registered [ 3347.036881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3350.309354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3350.318646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.497943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.269277] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3356.852928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.018478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3358.700268] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3361.670555] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3363.818543] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3366.778182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3366.958757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.333046] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3372.338273] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3376.176567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3381.217439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.221092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3381.223676] Lustre: Skipped 1 previous similar message [ 3381.231532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3382.923627] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 3382.927578] Lustre: Skipped 1 previous similar message [ 3386.204274] Lustre: server umount lustre-OST0000 complete [ 3389.261465] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa01d42afd800.ost_server_uuid 50 [ 3389.918511] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3400.072814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.073860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.078671] Lustre: Skipped 1 previous similar message [ 3406.233712] Lustre: server umount lustre-MDT0000 complete [ 3406.304065] LustreError: 188883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.312165] LustreError: 188883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3407.790414] LustreError: 188865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096238 with bad export cookie 2656922148978001653 [ 3407.793560] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.796100] LustreError: 188865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3407.930794] Lustre: server umount lustre-MDT0001 complete [ 3411.242529] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:17:21 (1784096241) [ 3413.890147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.091597] LustreError: 192774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.124420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.899193] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3419.520562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.616064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3419.619398] Lustre: Skipped 3 previous similar messages [ 3419.695896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.427421] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3424.504848] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3426.852202] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3429.611940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.743369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.084037] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3434.851201] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3437.497934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.579860] Lustre: lustre-OST0001: new disk, initializing [ 3437.583244] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3437.587546] Lustre: 195612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3438.795931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3438.801205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3438.834362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3439.843546] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3439.849784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3443.959352] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3445.217221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.220620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.226717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3449.318174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.321301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.147531] Lustre: server umount lustre-OST0000 complete [ 3453.356483] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3453.438775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3454.434915] LustreError: 194591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3454.440727] LustreError: 194591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3454.994672] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3455.080861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3459.557636] LustreError: 194594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.566670] LustreError: 194594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3464.672953] LustreError: 194594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3464.680199] LustreError: 194594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3469.798165] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3469.799041] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.803916] LustreError: Skipped 1 previous similar message [ 3469.804896] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3469.809717] Lustre: Skipped 1 previous similar message [ 3469.816245] Lustre: Skipped 4 previous similar messages [ 3471.333052] Lustre: server umount lustre-OST0001 complete [ 3487.199341] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3487.275235] Lustre: server umount lustre-MDT0000 complete [ 3488.816854] LustreError: 192754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096319 with bad export cookie 2656922148978002710 [ 3488.820469] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3488.822438] LustreError: 192754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3488.945525] Lustre: server umount lustre-MDT0001 complete [ 3498.512804] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:18:48 (1784096328) [ 3501.242763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.412131] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3501.430355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3501.518287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3501.521035] Lustre: Skipped 1 previous similar message [ 3503.092908] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3506.641830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.703646] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3506.728503] Lustre: 199027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3508.582580] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3511.627531] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3513.902649] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3516.738571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3516.826303] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3516.878045] LustreError: 200875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.888628] LustreError: 200875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3519.279334] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3522.022761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3522.230762] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3524.961378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3525.002609] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3527.485716] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3530.432399] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3532.256187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3532.259933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.265102] Lustre: Skipped 2 previous similar messages [ 3532.267485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3532.269994] Lustre: Skipped 6 previous similar messages [ 3537.605981] Lustre: server umount lustre-OST0000 complete [ 3540.450995] LustreError: 200875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.457140] LustreError: 200875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3550.690542] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.690664] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.693335] LustreError: Skipped 2 previous similar messages [ 3550.698311] Lustre: Skipped 1 previous similar message [ 3554.588275] Lustre: server umount lustre-OST0001 complete [ 3562.335780] Lustre: server umount lustre-MDT0000 complete [ 3563.850365] LustreError: 199773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096395 with bad export cookie 2656922148978004068 [ 3563.852209] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.856716] LustreError: 199773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3563.990514] Lustre: server umount lustre-MDT0001 complete [ 3573.516607] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:20:03 (1784096403) [ 3579.516823] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3583.106570] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3596.915032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.015823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3597.029754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3597.070150] Lustre: lustre-MDT0000: new disk, initializing [ 3597.095717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.098456] Lustre: Skipped 3 previous similar messages [ 3597.105200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3598.793594] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3603.641355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3603.703928] Lustre: 207089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3603.739688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3603.743041] Lustre: Skipped 1 previous similar message [ 3603.790754] Lustre: lustre-MDT0001: new disk, initializing [ 3603.850610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3603.855804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3605.612241] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3609.972738] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3612.347574] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3615.119327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3615.266610] Lustre: lustre-OST0000: new disk, initializing [ 3615.269953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3615.274084] Lustre: 209059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3617.895264] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3620.862819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3620.867829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3620.919362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3622.159842] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3623.927767] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3624.027207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3625.735250] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3625.831159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3631.073765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.074322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3631.075845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3631.075853] Lustre: Skipped 8 previous similar messages [ 3631.080324] Lustre: Skipped 4 previous similar messages [ 3631.084689] LustreError: Skipped 1 previous similar message [ 3632.734553] Lustre: server umount lustre-OST0000 complete [ 3640.421994] Lustre: server umount lustre-MDT0000 complete [ 3641.312271] LustreError: 207102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.319021] LustreError: 207102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3641.905376] LustreError: 207084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096473 with bad export cookie 2656922148978005573 [ 3641.908992] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.910059] LustreError: 207084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3642.039425] Lustre: server umount lustre-MDT0001 complete [ 3645.561342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.408198] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3650.621896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.355056] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3655.224533] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3657.361531] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3660.361379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.056820] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3665.901626] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3667.506696] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3671.694279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3673.369655] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3675.516218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 3680.227642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3680.231344] LustreError: Skipped 1 previous similar message [ 3682.396772] Lustre: server umount lustre-OST0000 complete [ 3684.826918] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3684.935216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3686.789502] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3711.611165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.812884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.817767] Lustre: Skipped 5 previous similar messages [ 3714.228280] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3717.062876] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3720.037292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3720.043310] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3720.043438] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3732.196425] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 14 sec [ 3735.524924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.526217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3735.530347] Lustre: Skipped 5 previous similar messages [ 3735.535431] Lustre: Skipped 7 previous similar messages [ 3739.541251] Lustre: server umount lustre-OST0000 complete [ 3747.229772] Lustre: server umount lustre-MDT0000 complete [ 3748.847695] LustreError: 211299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096580 with bad export cookie 2656922148978006455 [ 3748.851541] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3748.856662] LustreError: 211299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3748.982264] Lustre: server umount lustre-MDT0001 complete [ 3752.039815] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:23:02 (1784096582) [ 3754.546930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.743906] LustreError: 217316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3754.752476] LustreError: 217316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3756.493192] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3760.065664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.813189] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3764.606559] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3766.598428] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3768.953780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3771.438389] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3774.549715] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3776.381132] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3779.556086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3781.405713] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3781.520469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3784.310466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3784.411297] Lustre: lustre-OST0001: new disk, initializing [ 3784.413445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3784.416851] Lustre: 220703:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3787.003753] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3790.856754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3790.861108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3790.895309] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3791.390045] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.137346] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3793.243843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3794.968607] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3795.064633] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3795.940831] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3795.944369] LustreError: Skipped 3 previous similar messages [ 3797.911662] Lustre: server umount lustre-OST0001 complete [ 3800.937364] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3801.040466] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3802.782301] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3802.867364] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3826.919016] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3829.246264] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3831.831462] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3832.294802] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3832.301136] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3832.301204] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3832.308917] Lustre: Skipped 2 previous similar messages [ 3832.363916] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3835.070041] Lustre: server umount lustre-OST0001 complete [ 3853.394621] Lustre: server umount lustre-OST0000 complete [ 3860.961677] Lustre: server umount lustre-MDT0000 complete [ 3862.260121] LustreError: 217343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096693 with bad export cookie 2656922148978007407 [ 3862.263956] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3862.264432] LustreError: 217343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3868.478197] Lustre: server umount lustre-MDT0001 complete [ 3877.028142] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:25:07 (1784096707) [ 3879.382722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.506213] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3879.517538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3879.596577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.599182] Lustre: Skipped 5 previous similar messages [ 3881.135759] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3884.219774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.262228] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3884.273934] Lustre: 225933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3885.843773] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3888.311991] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3890.122323] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3892.391956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3892.474195] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3894.644834] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3896.991858] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3900.666506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3900.690533] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3902.780402] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3905.248092] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3906.758944] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3908.105801] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3908.186079] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3909.586635] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3909.662581] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3911.426725] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa01d602a6800.ost_server_uuid 50 [ 3911.933545] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa01d602a6800.ost_server_uuid in FULL state after 0 sec [ 3912.251670] Lustre: Permanently deactivating lustre-OST0001 [ 3912.256795] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3912.982788] Lustre: Permanently reactivating lustre-OST0001 [ 3916.770687] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3916.771959] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3916.776699] Lustre: Skipped 11 previous similar messages [ 3916.779137] Lustre: Skipped 11 previous similar messages [ 3919.997433] Lustre: server umount lustre-OST0001 complete [ 3921.889298] LustreError: 227776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.894407] LustreError: 227776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 3927.311073] Lustre: server umount lustre-OST0000 complete [ 3930.591812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3930.595369] LustreError: Skipped 7 previous similar messages [ 3934.752803] Lustre: server umount lustre-MDT0000 complete [ 3936.103875] LustreError: 225923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096767 with bad export cookie 2656922148978008681 [ 3936.106222] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3936.109125] LustreError: 225923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3936.246576] Lustre: server umount lustre-MDT0001 complete [ 3944.882315] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:26:15 (1784096775) [ 3947.500496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.611510] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3947.624421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3947.628229] Lustre: Skipped 5 previous similar messages [ 3949.105039] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3952.046896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.090290] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3952.105918] Lustre: 232618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3953.887416] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3956.771953] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3958.840318] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3961.458057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3961.582673] Lustre: Permanently deactivating lustre-OST0000 [ 3961.590660] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3963.787749] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3966.455083] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3966.969587] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3968.895077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3971.071630] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3973.082800] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3973.772369] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3974.406191] Lustre: Permanently reactivating lustre-OST0000 [ 3981.817595] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3981.821102] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3981.826514] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3981.848748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3981.858306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3990.659188] Lustre: server umount lustre-OST0001 complete [ 3998.297848] Lustre: server umount lustre-OST0000 complete [ 4005.980731] Lustre: server umount lustre-MDT0000 complete [ 4007.326016] LustreError: 232610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096838 with bad export cookie 2656922148978010319 [ 4007.329069] LustreError: 232610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4007.329095] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4007.460983] Lustre: server umount lustre-MDT0001 complete [ 4011.329525] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4012.473434] Key type lgssc unregistered [ 4012.608440] LNet: 237469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4012.611366] LNetError: 237469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4012.619283] LNet: Removed LNI 192.168.204.145@tcp [ 4012.960938] Key type .llcrypt unregistered [ 4012.962558] Key type ._llcrypt unregistered [ 4022.294485] Key type ._llcrypt registered [ 4022.296230] Key type .llcrypt registered [ 4022.349634] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4022.930768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4022.945748] alg: No test for adler32 (adler32-zlib) [ 4023.816256] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4023.925944] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4025.511178] Key type lgssc registered [ 4025.925328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4029.520834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4029.527450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.696225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4032.074792] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4035.117590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.169331] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4035.184938] Lustre: 238902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4035.192126] Lustre: Permanently deactivating lustre-MDT0001 [ 4035.198819] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4035.349182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4036.976587] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4039.849472] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4040.175293] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4040.179080] LustreError: 238943:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4040.184512] LustreError: 238943:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4040.189530] LustreError: 238943:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 1, failed: rc = -108 [ 4042.048374] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4044.700384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4044.850983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4045.859978] LustreError: 240751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.867043] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4045.868488] LustreError: 240751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4047.203421] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4050.145331] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4050.915990] LustreError: 240905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.688404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4052.764990] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4055.015601] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4055.854942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4055.854969] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4055.856352] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4057.644094] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4059.277332] Lustre: Permanently reactivating lustre-MDT0001 [ 4059.283336] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4059.285977] Lustre: Skipped 2 previous similar messages [ 4065.411731] Lustre: lustre-MDT0001: Client f010a223-8f56-4171-8f1b-c4be0f47b07a (at 192.168.204.45@tcp) reconnecting [ 4067.379610] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4067.917864] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4069.755633] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4069.829244] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4071.278605] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4071.360081] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4072.199675] Lustre: Permanently deactivating lustre-MDT0001 [ 4072.207625] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4072.210020] Lustre: Skipped 2 previous similar messages [ 4081.152266] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4086.754665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.756206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4086.756639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4086.758696] Lustre: Skipped 2 previous similar messages [ 4091.357283] Lustre: server umount lustre-MDT0000 complete [ 4091.873073] LustreError: 238911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.878880] LustreError: 238911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4092.643407] LustreError: 238892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784096923 with bad export cookie 13306147282497728739 [ 4092.647280] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4092.648322] LustreError: 238892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4092.760708] Lustre: server umount lustre-MDT0001 complete [ 4100.223450] Lustre: server umount lustre-OST0000 complete [ 4104.114148] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:28:54 (1784096934) [ 4108.018849] Lustre: server umount lustre-OST0001 complete [ 4110.311116] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4113.761441] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4125.724027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4125.812160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4125.815950] Lustre: Skipped 2 previous similar messages [ 4125.824793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4125.863872] Lustre: lustre-MDT0000: new disk, initializing [ 4125.886576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.894760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4127.316630] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4131.556056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.597552] Lustre: 247205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4131.626053] LustreError: 247217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4131.628868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4131.634067] Lustre: Skipped 1 previous similar message [ 4131.670280] Lustre: lustre-MDT0001: new disk, initializing [ 4131.715432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4131.719632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4133.062196] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4136.968115] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4138.949068] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4141.289863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4141.401328] Lustre: lustre-OST0000: new disk, initializing [ 4141.403424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4141.406992] Lustre: 249174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4141.436753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4141.438940] Lustre: Skipped 1 previous similar message [ 4143.424121] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4147.208868] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4150.773329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4150.777450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4150.807162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4152.920362] LustreError: 247213:0:(mdt_handler.c:3268:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4157.173650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4157.241546] Lustre: lustre-OST0001: new disk, initializing [ 4157.243272] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4157.245776] Lustre: 250291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4159.296412] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4163.015060] LustreError: 247213:0:(mdt_handler.c:3268:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4163.022674] LustreError: 247242:0:(mdt_handler.c:3268:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4163.209227] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4163.745842] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4165.119070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4165.122209] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4165.151277] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4173.119095] LustreError: 247242:0:(mdt_handler.c:3268:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4175.329510] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4175.331823] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.332503] LustreError: Skipped 1 previous similar message [ 4175.333525] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4175.336442] Lustre: Skipped 2 previous similar messages [ 4175.337895] Lustre: Skipped 3 previous similar messages [ 4180.454476] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4180.456408] Lustre: Skipped 2 previous similar messages [ 4185.571221] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4185.574365] Lustre: Skipped 2 previous similar messages [ 4188.127086] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4188.156072] Lustre: server umount lustre-OST0001 complete [ 4190.690673] LustreError: 249891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4190.696454] LustreError: 249891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4200.928922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.930280] LustreError: 249160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.930831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.930837] Lustre: Skipped 1 previous similar message [ 4200.934102] Lustre: Skipped 1 previous similar message [ 4200.939875] LustreError: 249160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4206.222308] Lustre: server umount lustre-OST0000 complete [ 4208.607863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4208.610767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.168480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4211.170513] Lustre: Skipped 5 previous similar messages [ 4213.722631] Lustre: server umount lustre-MDT0000 complete [ 4214.919242] LustreError: 250290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097046 with bad export cookie 13306147282497731546 [ 4214.921370] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4214.923675] LustreError: 250290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4221.105639] Lustre: server umount lustre-MDT0001 complete [ 4224.553627] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4225.588503] Key type lgssc unregistered [ 4225.736372] LNet: 252324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4225.739185] LNetError: 252324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4225.748380] LNet: Removed LNI 192.168.204.145@tcp [ 4226.060486] Key type .llcrypt unregistered [ 4226.061962] Key type ._llcrypt unregistered [ 4239.615451] Key type ._llcrypt registered [ 4239.616861] Key type .llcrypt registered [ 4239.656500] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4240.062309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4240.081784] alg: No test for adler32 (adler32-zlib) [ 4240.935690] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4241.027127] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4242.607167] Key type lgssc registered [ 4242.970249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4245.657348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4245.663345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.747939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4246.755680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4246.815736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4248.139776] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4250.964213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.004314] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4251.016109] Lustre: 254507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4251.110963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4252.586711] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4255.078702] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4256.993535] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4259.248067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4259.321313] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4259.403663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4261.088428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4261.362791] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4263.790869] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4307.425305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4307.428906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.428980] LustreError: Skipped 1 previous similar message [ 4307.434934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.438813] Lustre: Skipped 1 previous similar message [ 4308.706684] Lustre: server umount lustre-OST0000 complete [ 4310.276510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4314.369597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.483366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.418337] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4318.702400] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4327.779641] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.784413] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4327.784418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.790578] Lustre: Skipped 1 previous similar message [ 4353.505706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.506361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.508237] LustreError: Skipped 1 previous similar message [ 4353.508839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.512956] Lustre: Skipped 2 previous similar messages [ 4357.847142] Lustre: server umount lustre-OST0000 complete [ 4363.744822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4363.744888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.745437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.745441] Lustre: Skipped 1 previous similar message [ 4363.756057] Lustre: Skipped 1 previous similar message [ 4365.272046] Lustre: server umount lustre-MDT0000 complete [ 4366.447161] LustreError: 255248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097197 with bad export cookie 2051486975785595214 [ 4366.449579] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4366.450702] LustreError: 255248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4366.572246] Lustre: server umount lustre-MDT0001 complete [ 4370.093739] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4371.166670] Key type lgssc unregistered [ 4371.277560] LNet: 259916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4371.279960] LNetError: 259916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4371.288307] LNet: Removed LNI 192.168.204.145@tcp [ 4371.613376] Key type .llcrypt unregistered [ 4371.614387] Key type ._llcrypt unregistered [ 4380.359352] Key type ._llcrypt registered [ 4380.360334] Key type .llcrypt registered [ 4380.394474] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4380.813975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4380.823573] alg: No test for adler32 (adler32-zlib) [ 4381.666872] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4381.754081] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4383.335141] Key type lgssc registered [ 4383.715285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4386.408489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4386.414783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.576117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.857564] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4391.704950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.827637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4393.169717] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4395.553993] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4397.354639] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4399.616860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4399.749860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.760416] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4404.225104] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4412.071295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4422.625997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4422.626745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4422.628791] LustreError: Skipped 1 previous similar message [ 4422.634178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4422.637111] Lustre: Skipped 1 previous similar message [ 4427.746396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4427.748543] Lustre: Skipped 1 previous similar message [ 4427.912750] Lustre: server umount lustre-OST0000 complete [ 4432.865504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4432.865935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.870490] Lustre: Skipped 1 previous similar message [ 4432.872783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.875312] Lustre: Skipped 1 previous similar message [ 4435.284884] Lustre: server umount lustre-MDT0000 complete [ 4436.508169] LustreError: 261291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097267 with bad export cookie 1182148350738528811 [ 4436.510852] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4436.511780] LustreError: 261291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4436.629339] Lustre: server umount lustre-MDT0001 complete [ 4440.227858] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4441.181970] Key type lgssc unregistered [ 4441.316387] LNet: 265633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4441.318676] LNetError: 265633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4441.328297] LNet: Removed LNI 192.168.204.145@tcp [ 4441.615249] Key type .llcrypt unregistered [ 4441.616766] Key type ._llcrypt unregistered [ 4448.644830] Key type ._llcrypt registered [ 4448.646431] Key type .llcrypt registered [ 4448.683925] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4449.058375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4449.112545] alg: No test for adler32 (adler32-zlib) [ 4449.981377] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4450.073206] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4451.655163] Key type lgssc registered [ 4452.021427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4454.380432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4454.386626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.513568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4456.813969] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4459.713861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4459.840262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4461.261955] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4463.812755] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4465.740644] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4468.007912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4468.119088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.054824] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4472.417871] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4480.421299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4489.696026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4489.698577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.703177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4490.720840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4490.723436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.730423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4492.820597] Lustre: server umount lustre-OST0000 complete [ 4495.841510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4495.841891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4495.842654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4495.842660] Lustre: Skipped 1 previous similar message [ 4495.852638] Lustre: Skipped 1 previous similar message [ 4500.191411] Lustre: server umount lustre-MDT0000 complete [ 4500.959949] LustreError: 268256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.965063] LustreError: 268256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4501.445154] LustreError: 269488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097332 with bad export cookie 13309494613980788502 [ 4501.448581] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.449687] LustreError: 269488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4501.558906] Lustre: server umount lustre-MDT0001 complete [ 4505.164779] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4506.150790] Key type lgssc unregistered [ 4506.268448] LNet: 270526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4506.271188] LNetError: 270526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4506.279324] LNet: Removed LNI 192.168.204.145@tcp [ 4506.566425] Key type .llcrypt unregistered [ 4506.567984] Key type ._llcrypt unregistered [ 4515.673355] Key type ._llcrypt registered [ 4515.674774] Key type .llcrypt registered [ 4515.712565] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4516.055523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4516.089085] alg: No test for adler32 (adler32-zlib) [ 4516.940526] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4517.037572] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4518.623194] Key type lgssc registered [ 4518.973932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4521.517266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4521.523808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4522.664611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.973739] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4526.790200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.912379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4528.258451] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4530.708600] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4532.538242] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4534.787615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4534.915205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4536.756412] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4539.035437] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4547.238112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4557.793942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4557.794329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.796517] LustreError: Skipped 1 previous similar message [ 4557.801183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4557.803461] Lustre: Skipped 1 previous similar message [ 4558.990758] Lustre: server umount lustre-OST0000 complete [ 4562.323832] Lustre: server umount lustre-MDT0000 complete [ 4562.912024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4562.913434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.915055] LustreError: 271918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.920153] Lustre: Skipped 2 previous similar messages [ 4562.927635] LustreError: 271918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4563.521287] LustreError: 271900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097394 with bad export cookie 451035832228530105 [ 4563.523809] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4563.524983] LustreError: 271900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4563.631692] Lustre: server umount lustre-MDT0001 complete [ 4567.098916] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4568.078021] Key type lgssc unregistered [ 4568.218356] LNet: 276237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4568.221442] LNetError: 276237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4568.230440] LNet: Removed LNI 192.168.204.145@tcp [ 4568.538340] Key type .llcrypt unregistered [ 4568.539601] Key type ._llcrypt unregistered [ 4575.798421] Key type ._llcrypt registered [ 4575.799947] Key type .llcrypt registered [ 4575.839120] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4576.255357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4576.264829] alg: No test for adler32 (adler32-zlib) [ 4577.110692] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4577.193369] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4578.775130] Key type lgssc registered [ 4579.145894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4581.715550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4581.723235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.876893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.206562] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4587.070152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.203786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4588.597982] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4590.954554] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4592.755267] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4594.939503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4595.038957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.747516] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4599.011737] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4608.357609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4613.602417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.603049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4613.603208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4613.606222] Lustre: Skipped 1 previous similar message [ 4615.178751] Lustre: server umount lustre-OST0000 complete [ 4618.207836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4618.210258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4618.214296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.215961] Lustre: Skipped 1 previous similar message [ 4618.447773] Lustre: server umount lustre-MDT0000 complete [ 4618.720968] LustreError: 277443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.726147] LustreError: 277443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4619.659226] LustreError: 277416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097450 with bad export cookie 11661307337223077537 [ 4619.662215] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4619.663273] LustreError: 277416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4619.770844] Lustre: server umount lustre-MDT0001 complete [ 4623.204061] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4624.151961] Key type lgssc unregistered [ 4624.285386] LNet: 281166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4624.287762] LNetError: 281166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4624.294241] LNet: Removed LNI 192.168.204.145@tcp [ 4624.584857] Key type .llcrypt unregistered [ 4624.585834] Key type ._llcrypt unregistered [ 4627.896987] Key type ._llcrypt registered [ 4627.898311] Key type .llcrypt registered [ 4627.932708] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4633.028893] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4638.953868] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4639.284412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4639.344274] alg: No test for adler32 (adler32-zlib) [ 4640.191981] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4640.281546] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4641.863212] Key type lgssc registered [ 4642.230054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4653.494608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4653.501101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.580745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4654.591127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4654.626707] Lustre: lustre-MDT0000: new disk, initializing [ 4654.649816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.656112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4656.045514] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4660.123022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.166865] Lustre: 284687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4660.189904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4660.192131] Lustre: Skipped 1 previous similar message [ 4660.224861] Lustre: lustre-MDT0001: new disk, initializing [ 4660.257968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4660.271095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4660.274355] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4661.596182] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4665.472199] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4667.373763] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4669.667164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4669.785089] Lustre: lustre-OST0000: new disk, initializing [ 4669.787529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4669.791205] Lustre: 286654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4669.825202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.843113] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4671.997481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4672.000921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4672.045327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4675.567356] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4676.922171] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4677.001196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4678.440131] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4678.519777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4682.212347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4682.212455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.214446] LustreError: Skipped 1 previous similar message [ 4682.218025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4682.220672] Lustre: Skipped 1 previous similar message [ 4685.264727] Lustre: server umount lustre-OST0000 complete [ 4687.329510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.330305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4687.333977] Lustre: Skipped 2 previous similar messages [ 4692.448034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.449862] Lustre: Skipped 2 previous similar messages [ 4692.626000] Lustre: server umount lustre-MDT0000 complete [ 4693.792020] LustreError: 284681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097524 with bad export cookie 10122715029461416156 [ 4693.794907] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.796074] LustreError: 284681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4693.909446] Lustre: server umount lustre-MDT0001 complete [ 4696.392938] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:38:46 (1784097526) [ 4698.402069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.582558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.887744] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4702.477054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.593233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4703.806012] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4705.948654] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4707.535934] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4709.543198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4711.476493] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4713.728144] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4728.288657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4728.291070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.295865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4729.314043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4733.132535] Lustre: server umount lustre-OST0000 complete [ 4734.432184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4734.432651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4734.435563] Lustre: Skipped 2 previous similar messages [ 4734.437816] Lustre: Skipped 2 previous similar messages [ 4740.439419] Lustre: server umount lustre-MDT0000 complete [ 4741.713310] LustreError: 289050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097572 with bad export cookie 10122715029461417024 [ 4741.715537] LustreError: 289806:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4741.716281] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4741.718038] LustreError: 289050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4741.839309] Lustre: server umount lustre-MDT0001 complete [ 4745.387410] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4746.456524] Key type lgssc unregistered [ 4746.588679] LNet: 292622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4746.591280] LNetError: 292622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4746.598282] LNet: Removed LNI 192.168.204.145@tcp [ 4746.906647] Key type .llcrypt unregistered [ 4746.908322] Key type ._llcrypt unregistered [ 4755.180250] Key type ._llcrypt registered [ 4755.181480] Key type .llcrypt registered [ 4755.216017] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4760.992101] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4761.320206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4761.386373] alg: No test for adler32 (adler32-zlib) [ 4762.257654] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4762.353824] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4763.943156] Key type lgssc registered [ 4764.292985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4777.849032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4777.856314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.925683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4778.933759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4778.966984] Lustre: lustre-MDT0000: new disk, initializing [ 4778.985555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.990775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4780.205167] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4784.041889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.081451] Lustre: 296418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4784.105805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4784.108071] Lustre: Skipped 1 previous similar message [ 4784.142915] Lustre: lustre-MDT0001: new disk, initializing [ 4784.171779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4784.185127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4784.189488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4785.473831] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4789.053382] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4790.866607] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4793.108935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.218724] Lustre: lustre-OST2710: new disk, initializing [ 4793.220389] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4793.222530] Lustre: 298383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4793.251415] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4793.629850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4793.633879] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4793.657459] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4794.953445] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4798.583850] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4800.988536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.044318] Lustre: lustre-OST03e8: new disk, initializing [ 4801.045980] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4801.048238] Lustre: 299445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4801.077036] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4802.833135] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4803.643566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4803.646978] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4803.673734] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4805.136983] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4807.712091] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4807.770290] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4809.050808] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4809.121320] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4810.336194] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4810.403117] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4811.798197] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4811.880786] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4813.795177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.796581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4813.798934] Lustre: Skipped 3 previous similar messages [ 4813.800506] Lustre: Skipped 2 previous similar messages [ 4818.913659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4818.916580] Lustre: Skipped 3 previous similar messages [ 4819.105335] Lustre: server umount lustre-MDT0000 complete [ 4820.298981] LustreError: 296408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097651 with bad export cookie 12919333404597031041 [ 4820.302243] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4820.303863] LustreError: 296408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4820.405472] Lustre: server umount lustre-MDT0001 complete [ 4831.232255] Lustre: server umount lustre-OST2710 complete [ 4842.591959] Lustre: server umount lustre-OST03e8 complete [ 4848.130685] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4851.076185] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4861.933978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.013616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4862.023949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4862.064374] Lustre: lustre-MDT0000: new disk, initializing [ 4862.083779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.089573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4863.340407] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4867.225626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.261551] Lustre: 304686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4867.356681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4868.639958] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4872.221660] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4873.916355] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4875.962759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4876.050115] Lustre: lustre-OST0000: new disk, initializing [ 4876.051546] Lustre: Skipped 1 previous similar message [ 4876.052805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4876.054294] Lustre: Skipped 2 previous similar messages [ 4876.056031] Lustre: 306651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4876.080404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4876.083209] Lustre: Skipped 1 previous similar message [ 4877.856805] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4878.516370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4878.519591] Lustre: Skipped 1 previous similar message [ 4878.521208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4878.545258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4881.315535] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4882.568545] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4882.634292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4883.840804] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4883.904019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4889.056992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4889.057994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.059430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4889.060089] LustreError: Skipped 1 previous similar message [ 4889.064481] Lustre: Skipped 1 previous similar message [ 4890.631116] Lustre: server umount lustre-OST0000 complete [ 4893.151749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4893.154624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.160703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.162927] Lustre: Skipped 1 previous similar message [ 4897.882766] Lustre: server umount lustre-MDT0000 complete [ 4899.187073] LustreError: 304678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097730 with bad export cookie 12919333404597032707 [ 4899.190274] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4899.190676] LustreError: 304678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4899.316829] Lustre: server umount lustre-MDT0001 complete [ 4902.226796] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:42:12 (1784097732) [ 4902.753337] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4903.348180] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:42:13 (1784097733) [ 4904.420091] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4914.028354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4914.108341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4914.114829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4914.171101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.588472] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4918.639311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.676574] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4918.687444] Lustre: 310500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4920.218436] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4936.226102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4936.285221] Lustre: Denying initial registration attempt from nid 192.168.204.145@tcp, specified as failover [ 4936.288359] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4936.291266] LustreError: 311955:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4936.294519] LustreError: 311955:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4936.296723] LustreError: 311955:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4936.319280] Lustre: server umount lustre-OST0000 complete [ 4936.320547] LustreError: 311955:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4939.232035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4939.234182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.238596] Lustre: Skipped 1 previous similar message [ 4939.240359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4939.242152] Lustre: Skipped 2 previous similar messages [ 4944.657331] Lustre: server umount lustre-MDT0000 complete [ 4945.946303] LustreError: 310492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097777 with bad export cookie 12919333404597033575 [ 4945.949745] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4945.952658] LustreError: 310492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4946.074716] Lustre: server umount lustre-MDT0001 complete [ 4949.873912] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4950.878604] Key type lgssc unregistered [ 4950.999341] LNet: 313073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4951.001824] LNetError: 313073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4951.010304] LNet: Removed LNI 192.168.204.145@tcp [ 4951.294534] Key type .llcrypt unregistered [ 4951.295513] Key type ._llcrypt unregistered [ 4954.784847] Key type ._llcrypt registered [ 4954.785822] Key type .llcrypt registered [ 4954.819925] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4955.180778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4955.197566] alg: No test for adler32 (adler32-zlib) [ 4956.049689] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 4956.128728] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4957.703153] Key type lgssc registered [ 4958.051845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4982.264257] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4985.003656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4985.009182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.077274] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4986.083659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4986.132587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4987.391075] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4990.405474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.440861] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4990.454211] Lustre: 315809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4990.557133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4992.056682] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4994.664493] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4996.766578] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5012.322070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5012.392286] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5012.465577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5014.573853] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5017.236694] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5017.975724] Lustre: server umount lustre-OST0000 complete [ 5021.152095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5021.156277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.162461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.190065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.190522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.361564] Lustre: server umount lustre-MDT0000 complete [ 5026.543129] LustreError: 315802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097857 with bad export cookie 12471037162254023259 [ 5026.546067] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5026.547545] LustreError: 315802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5026.650878] Lustre: server umount lustre-MDT0001 complete [ 5030.349961] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5031.299472] Key type lgssc unregistered [ 5031.424298] LNet: 319375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5031.427216] LNetError: 319375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5031.435277] LNet: Removed LNI 192.168.204.145@tcp [ 5031.715424] Key type .llcrypt unregistered [ 5031.716927] Key type ._llcrypt unregistered [ 5041.138130] Key type ._llcrypt registered [ 5041.139263] Key type .llcrypt registered [ 5041.178667] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5041.630533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5041.639437] alg: No test for adler32 (adler32-zlib) [ 5042.501919] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 5042.600674] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5044.183168] Key type lgssc registered [ 5044.605767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.368805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5047.375349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.524240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5049.920825] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5052.771308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.900202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5054.242499] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5056.823080] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5058.750519] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5062.817603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5062.915331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.653106] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5067.068473] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5082.760758] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 5083.617036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5083.617850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.618415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5083.628314] Lustre: Skipped 2 previous similar messages [ 5085.972994] Lustre: server umount lustre-MDT0000 complete [ 5087.207075] LustreError: 320747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097918 with bad export cookie 16705583771960324180 [ 5087.209927] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5087.212043] LustreError: 320747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5087.869884] LustreError: 322605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5087.875095] LustreError: 322605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5087.876083] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 5087.880046] Lustre: Skipped 2 previous similar messages [ 5088.735718] LustreError: 320762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.735880] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.767719] LustreError: 322605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5092.767906] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5092.778572] Lustre: Skipped 1 previous similar message [ 5093.346723] Lustre: server umount lustre-MDT0001 complete [ 5095.376879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5098.337260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.287845] LustreError: 320078:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000222a7850 x1870762211303168/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5112.427232] LustreError: 322595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.432619] LustreError: 322595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5112.457957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.484899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5113.949048] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5117.192659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.326192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5118.372801] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5118.772340] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5121.435866] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5122.467312] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5125.054529] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5126.972741] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5127.575516] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5132.769619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5132.770126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.773122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.775903] Lustre: Skipped 1 previous similar message [ 5132.778536] Lustre: Skipped 1 previous similar message [ 5134.991609] Lustre: server umount lustre-OST0000 complete [ 5137.888323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5137.888845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5137.896738] Lustre: Skipped 1 previous similar message [ 5142.289935] Lustre: server umount lustre-MDT0000 complete [ 5143.008914] LustreError: 324377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5143.014256] LustreError: 324377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5143.427253] LustreError: 320747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784097974 with bad export cookie 16705583771960331334 [ 5143.429682] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5143.430971] LustreError: 320747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5143.536208] Lustre: server umount lustre-MDT0001 complete [ 5147.151659] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5148.118839] Key type lgssc unregistered [ 5148.261404] LNet: 327042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5148.263803] LNetError: 327042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5148.271385] LNet: Removed LNI 192.168.204.145@tcp [ 5148.545975] Key type .llcrypt unregistered [ 5148.547080] Key type ._llcrypt unregistered [ 5157.385987] Key type ._llcrypt registered [ 5157.387222] Key type .llcrypt registered [ 5157.424173] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5157.821569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5157.827936] alg: No test for adler32 (adler32-zlib) [ 5158.672876] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 5158.747710] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5160.319156] Key type lgssc registered [ 5160.632836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5163.029365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5163.036346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.199703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.431733] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5168.077584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.193813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5169.478376] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5171.612305] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5173.291636] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5178.336389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5178.337324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.337734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5178.345371] Lustre: Skipped 1 previous similar message [ 5180.372624] Lustre: server umount lustre-MDT0000 complete [ 5181.576762] LustreError: 328415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098012 with bad export cookie 14366486943176611251 [ 5181.581817] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.581932] LustreError: 328415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5181.696372] Lustre: server umount lustre-MDT0001 complete [ 5186.197697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.282992] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5186.291867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5186.355230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.513554] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5190.342752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.375148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5190.385411] Lustre: 331016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5190.490218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5191.936832] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5194.403447] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5196.348554] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5200.993914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5201.061872] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5201.065932] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5201.068741] LustreError: 332900:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5201.071842] LustreError: 332900:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5201.074666] LustreError: 332900:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5201.099802] Lustre: server umount lustre-OST0000 complete [ 5201.101251] LustreError: 332900:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5205.988134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5206.052862] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5206.118979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.076755] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5210.626480] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5213.284317] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5213.320985] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5213.351575] Lustre: lustre-OST0001: new disk, initializing [ 5213.353288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5213.355706] Lustre: 334429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5214.991449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5214.995462] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5215.030670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5215.057988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5215.587065] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5218.221403] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5220.324577] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.327932] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5220.329874] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5220.331237] LustreError: Skipped 1 previous similar message [ 5220.333250] Lustre: Skipped 1 previous similar message [ 5221.247565] Lustre: server umount lustre-OST0001 complete [ 5225.441218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5225.442036] LustreError: 335236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.443307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.443315] Lustre: Skipped 1 previous similar message [ 5225.444152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5225.448740] LustreError: 335236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5225.452630] Lustre: Skipped 1 previous similar message [ 5228.493103] Lustre: server umount lustre-OST0000 complete [ 5230.560864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.561442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5230.565879] Lustre: Skipped 2 previous similar messages [ 5235.680574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5235.682166] Lustre: Skipped 2 previous similar messages [ 5235.798632] Lustre: server umount lustre-MDT0000 complete [ 5236.979921] LustreError: 331006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098068 with bad export cookie 14366486943176611790 [ 5236.982907] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5236.983255] LustreError: 331006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5237.087167] Lustre: server umount lustre-MDT0001 complete [ 5240.459609] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5241.392154] Key type lgssc unregistered [ 5241.507292] LNet: 336390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5241.509443] LNetError: 336390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5241.518270] LNet: Removed LNI 192.168.204.145@tcp [ 5241.811191] Key type .llcrypt unregistered [ 5241.812274] Key type ._llcrypt unregistered [ 5271.228807] Key type ._llcrypt registered [ 5271.229734] Key type .llcrypt registered [ 5271.261937] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5277.260168] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5277.677903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5277.707104] alg: No test for adler32 (adler32-zlib) [ 5278.556969] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 5278.639686] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5280.215170] Key type lgssc registered [ 5280.576512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5291.416477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5291.421526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5292.513423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5292.524371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5292.564058] Lustre: lustre-MDT0000: new disk, initializing [ 5292.583790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5292.590714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5293.884225] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5297.851773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.893602] Lustre: 341297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5297.917785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5297.920797] Lustre: Skipped 1 previous similar message [ 5297.966141] Lustre: lustre-MDT0001: new disk, initializing [ 5298.003372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5298.017392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5298.021287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5299.312775] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5302.938506] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5304.737112] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5306.917029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5307.028535] Lustre: lustre-OST0000: new disk, initializing [ 5307.030654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5307.034071] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5307.034387] Lustre: 343265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5307.072641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.911262] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5312.490503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5312.495875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5312.510684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5312.631932] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5314.152734] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5314.230811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5315.582772] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5315.663152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5317.600657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5317.602855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.603948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5317.608066] Lustre: Skipped 1 previous similar message [ 5318.427049] Lustre: server umount lustre-OST0000 complete [ 5322.720566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.721215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5322.724589] Lustre: Skipped 1 previous similar message [ 5322.729018] Lustre: Skipped 2 previous similar messages [ 5327.840114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5327.843534] Lustre: Skipped 1 previous similar message [ 5332.960084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.962523] Lustre: Skipped 1 previous similar message [ 5333.983088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5334.044304] Lustre: server umount lustre-MDT0000 complete [ 5335.207512] LustreError: 343263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098166 with bad export cookie 14424498350999426046 [ 5335.212344] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5335.324359] Lustre: server umount lustre-MDT0001 complete [ 5337.859846] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:49:28 (1784098168) [ 5340.193844] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:49:30 (1784098170) [ 5342.741174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.907817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5344.220342] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5346.857866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.956638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5348.257871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5350.536282] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5352.279974] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5354.423435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5356.450246] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5358.795306] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5372.747668] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5373.247384] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5373.734441] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5374.242545] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5374.826881] Lustre: Failing over lustre-MDT0000 [ 5375.013256] Lustre: server umount lustre-MDT0000 complete [ 5378.019079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.020588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5378.021630] LustreError: 346021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.021660] LustreError: 346021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5378.023951] Lustre: Skipped 1 previous similar message [ 5378.034319] LustreError: Skipped 1 previous similar message [ 5379.718713] LustreError: 346022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.723815] LustreError: 346022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5383.137754] LustreError: 346025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.145209] LustreError: 346025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5387.942171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.986556] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5388.091618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5388.093724] Lustre: Skipped 1 previous similar message [ 5388.123341] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5389.526329] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5389.949173] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5393.383220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5393.392584] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5393.408114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5395.285431] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5395.748947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5397.565636] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5398.187330] Lustre: Failing over lustre-MDT0000 [ 5398.412534] Lustre: server umount lustre-MDT0000 complete [ 5398.496935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5398.497890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.498346] LustreError: 346026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.498352] LustreError: 346026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5398.508197] Lustre: Skipped 3 previous similar messages [ 5403.616255] LustreError: 347508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.620863] LustreError: 347508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5411.140504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.182573] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5411.277958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.304691] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5412.653511] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5415.549148] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5416.422961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5416.425148] Lustre: Skipped 2 previous similar messages [ 5416.433812] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5416.451645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5418.269206] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5418.759835] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5421.538268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.538713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5421.539359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.542180] Lustre: Skipped 2 previous similar messages [ 5427.157275] Lustre: server umount lustre-MDT0000 complete [ 5428.315048] LustreError: 346006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098259 with bad export cookie 14424498350999428727 [ 5428.318320] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5428.319895] LustreError: 346006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5428.414161] Lustre: server umount lustre-MDT0001 complete [ 5439.826982] Lustre: server umount lustre-OST0000 complete [ 5444.577443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.706995] LustreError: 352376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.712893] LustreError: 352376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5444.727165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5445.974077] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5448.647097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.074658] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5452.311963] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5453.987136] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5456.106216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5457.996440] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5460.259022] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5469.540472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5473.759889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5473.762257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.766201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5473.768707] Lustre: Skipped 5 previous similar messages [ 5478.224494] Lustre: server umount lustre-OST0000 complete [ 5484.511943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5484.514161] LustreError: Skipped 1 previous similar message [ 5485.646300] Lustre: server umount lustre-MDT0000 complete [ 5486.890256] LustreError: 353110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098318 with bad export cookie 14424498350999429448 [ 5486.893332] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5486.897209] LustreError: 353110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5487.024473] Lustre: server umount lustre-MDT0001 complete [ 5490.604637] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5491.540602] Key type lgssc unregistered [ 5491.656454] LNet: 355925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5491.659336] LNetError: 355925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5491.670526] LNet: Removed LNI 192.168.204.145@tcp [ 5492.013956] Key type .llcrypt unregistered [ 5492.015073] Key type ._llcrypt unregistered [ 5501.941504] Key type ._llcrypt registered [ 5501.942535] Key type .llcrypt registered [ 5501.976456] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5502.318416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5502.376132] alg: No test for adler32 (adler32-zlib) [ 5503.220153] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 5503.299745] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5504.871132] Key type lgssc registered [ 5505.183860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5507.750824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5507.756782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.923519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.331302] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5513.037939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.154490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5514.405080] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5516.599431] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5518.346259] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5520.534534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5520.639722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5522.529660] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5524.874119] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5533.990839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5539.297159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5539.298018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5539.301490] Lustre: Skipped 2 previous similar messages [ 5539.303567] Lustre: Skipped 1 previous similar message [ 5544.416092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5544.418082] Lustre: Skipped 2 previous similar messages [ 5545.236661] Lustre: server umount lustre-MDT0000 complete [ 5546.379923] LustreError: 357400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098377 with bad export cookie 9682896615515648616 [ 5546.383263] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5546.383661] LustreError: 357400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5546.477952] Lustre: server umount lustre-MDT0001 complete [ 5556.950954] Lustre: server umount lustre-OST0000 complete [ 5561.326856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.474235] LustreError: 361145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5561.480425] LustreError: 361145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5561.494539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5562.681752] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5565.294190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.643989] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5568.893241] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5570.553119] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5572.658420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5572.778619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.780456] Lustre: Skipped 1 previous similar message [ 5574.617486] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5576.834445] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5586.086681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5596.127749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5596.130168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.134191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5600.462081] Lustre: server umount lustre-MDT0000 complete [ 5601.571983] LustreError: 361125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098432 with bad export cookie 9682896615515649722 [ 5601.574918] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5601.575532] LustreError: 361125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5601.673502] Lustre: server umount lustre-MDT0001 complete [ 5613.083078] Lustre: server umount lustre-OST0000 complete [ 5618.156030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.321571] LustreError: 365078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.345516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.600229] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5622.256469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.609715] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5625.908512] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5627.651400] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5629.684017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5631.654386] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5634.061151] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5644.134210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5647.327808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.330495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.333235] Lustre: Skipped 2 previous similar messages [ 5647.334579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5647.336474] Lustre: Skipped 2 previous similar messages [ 5649.377302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5649.378453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.595877] Lustre: server umount lustre-OST0000 complete [ 5652.959928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5652.962868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.965006] Lustre: Skipped 2 previous similar messages [ 5654.496790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.499889] Lustre: Skipped 1 previous similar message [ 5664.736204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5664.737898] Lustre: Skipped 5 previous similar messages [ 5667.295117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5667.355921] Lustre: server umount lustre-MDT0000 complete [ 5668.578477] LustreError: 367576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098499 with bad export cookie 9682896615515650730 [ 5668.581567] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5668.583721] LustreError: 367576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5668.699680] Lustre: server umount lustre-MDT0001 complete [ 5672.252938] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5673.203739] Key type lgssc unregistered [ 5673.329300] LNet: 368629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5673.332791] LNetError: 368629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5673.341280] LNet: Removed LNI 192.168.204.145@tcp [ 5673.635288] Key type .llcrypt unregistered [ 5673.636683] Key type ._llcrypt unregistered [ 5682.682903] Key type ._llcrypt registered [ 5682.684313] Key type .llcrypt registered [ 5682.723686] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5683.103038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5683.124386] alg: No test for adler32 (adler32-zlib) [ 5683.982984] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 5684.076321] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5685.663179] Key type lgssc registered [ 5686.018958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5688.349413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5688.355535] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.358217] LustreError: 370046:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5688.366800] LustreError: 370046:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5688.368887] LustreError: 370046:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5688.371104] LustreError: 370046:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5688.373408] LustreError: 370046:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5690.652689] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5690.655182] LustreError: 370443:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5690.663860] LustreError: 370443:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5690.666045] LustreError: 370443:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5690.668620] LustreError: 370443:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5690.671419] LustreError: 370443:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5696.704518] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5697.737251] Key type lgssc unregistered [ 5697.864341] LNet: 371445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5697.866512] LNetError: 371445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5697.877300] LNet: Removed LNI 192.168.204.145@tcp [ 5698.168450] Key type .llcrypt unregistered [ 5698.169485] Key type ._llcrypt unregistered [ 5703.799277] Key type ._llcrypt registered [ 5703.800529] Key type .llcrypt registered [ 5703.843117] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5710.078362] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5710.398284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5710.459665] alg: No test for adler32 (adler32-zlib) [ 5711.320213] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 5711.406918] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5712.983143] Key type lgssc registered [ 5713.314826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5724.176369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5724.180712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5725.270170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5725.280822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5725.320258] Lustre: lustre-MDT0000: new disk, initializing [ 5725.340408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5725.347570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5726.539114] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5730.365443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.406409] Lustre: 374574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5730.428513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5730.430949] Lustre: Skipped 1 previous similar message [ 5730.469532] Lustre: lustre-MDT0001: new disk, initializing [ 5730.500021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5730.511449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5730.515422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5731.706739] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5735.278212] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5737.113877] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5739.340288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5739.438704] Lustre: lustre-OST0000: new disk, initializing [ 5739.440541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5739.443142] Lustre: 376539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5739.470660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.411329] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5743.414391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5743.419126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5743.450214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5745.177525] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5746.440221] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5746.505353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5747.763586] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5747.826414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5748.704993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5748.705677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5748.709572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5748.713968] Lustre: Skipped 1 previous similar message [ 5750.483072] Lustre: server umount lustre-OST0000 complete [ 5753.825569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.827875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.830014] Lustre: Skipped 2 previous similar messages [ 5753.832954] Lustre: Skipped 1 previous similar message [ 5757.851269] Lustre: server umount lustre-MDT0000 complete [ 5759.054136] LustreError: 374564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098590 with bad export cookie 8822492065330018719 [ 5759.055824] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.058786] LustreError: 374564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5759.167244] Lustre: server umount lustre-MDT0001 complete [ 5761.733342] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5762.264618] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:56:32 (1784098592) [ 5764.390800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5764.535568] LustreError: 378995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.541928] LustreError: 378995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5764.558243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.856962] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5768.481481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.591491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5769.838589] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5772.112149] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5773.855796] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5775.982373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5777.826137] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5780.120793] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5782.288260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.344327] Lustre: lustre-OST0001: new disk, initializing [ 5782.346240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5782.348660] Lustre: 381829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5782.374557] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5782.376724] Lustre: Skipped 1 previous similar message [ 5784.226272] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5784.509028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5784.512273] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5784.539524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5787.929263] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5794.438399] Lustre: lustre-OST0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 5794.788279] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5794.788488] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.790630] LustreError: Skipped 1 previous similar message [ 5795.843990] Lustre: server umount lustre-OST0001 complete [ 5799.907228] LustreError: 380815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.912402] LustreError: 380815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5805.027403] LustreError: 380815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.034100] LustreError: 380815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5809.119815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5809.122756] LustreError: Skipped 1 previous similar message [ 5809.124371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.127787] Lustre: Skipped 1 previous similar message [ 5809.129735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5809.131765] Lustre: Skipped 2 previous similar messages [ 5810.147639] LustreError: 382231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5810.153290] LustreError: 382231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5813.961705] Lustre: server umount lustre-OST0000 complete [ 5815.266270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5815.270857] Lustre: Skipped 1 previous similar message [ 5815.272819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5815.275085] Lustre: Skipped 2 previous similar messages [ 5821.591485] Lustre: server umount lustre-MDT0000 complete [ 5822.757123] LustreError: 378976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098653 with bad export cookie 8822492065330019587 [ 5822.759399] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5822.761237] LustreError: 378976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5837.279152] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5837.363701] Lustre: server umount lustre-MDT0001 complete [ 5840.695865] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5841.758950] Key type lgssc unregistered [ 5841.897452] LNet: 383814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5841.900493] LNetError: 383814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5841.915320] LNet: Removed LNI 192.168.204.145@tcp [ 5842.253838] Key type .llcrypt unregistered [ 5842.254899] Key type ._llcrypt unregistered [ 5859.752837] Key type ._llcrypt registered [ 5859.753953] Key type .llcrypt registered [ 5859.791407] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5860.173317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5860.224054] alg: No test for adler32 (adler32-zlib) [ 5861.093507] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 5861.191939] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5862.775263] Key type lgssc registered [ 5863.140606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5865.706505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5865.714348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.801578] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5866.810975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5866.878203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.334617] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5871.255567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.288698] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5871.300529] Lustre: 386587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5871.411705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5872.810273] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5875.251420] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5876.986679] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5879.264564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5879.265126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5879.269097] Lustre: Skipped 1 previous similar message [ 5883.729423] Lustre: server umount lustre-MDT0000 complete [ 5884.915405] LustreError: 386577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098716 with bad export cookie 2928479849443375724 [ 5884.918435] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.919447] LustreError: 386577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5885.033351] Lustre: server umount lustre-MDT0001 complete [ 5887.923636] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:58:38 (1784098718) [ 5889.991313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5890.130608] LustreError: 389097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5890.136445] LustreError: 389097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5890.146676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5891.420846] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5894.084195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.195536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5895.477760] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5897.752120] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5899.471634] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5901.696225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.772541] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5901.827206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.621107] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5905.883067] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5909.701621] Lustre: Permanently deactivating lustre-OST0000 [ 5909.706667] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5913.517834] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5914.060431] LustreError: 392033:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5914.064899] LustreError: 392033:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5914.067379] LustreError: 392033:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5914.679911] LustreError: 392082:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5914.682223] LustreError: 392082:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5921.424682] Lustre: server umount lustre-OST0000 complete [ 5922.351867] LustreError: 392332:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5922.355743] LustreError: 392332:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5924.832040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5924.834751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.838792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5924.841272] Lustre: Skipped 1 previous similar message [ 5925.856648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5925.857374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.976074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5930.978387] Lustre: Skipped 2 previous similar messages [ 5933.046975] Lustre: server umount lustre-MDT0000 complete [ 5934.266158] LustreError: 389077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098765 with bad export cookie 2928479849443376284 [ 5934.269434] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5934.270129] LustreError: 389077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5944.852912] Lustre: server umount lustre-MDT0001 complete [ 5947.797981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.283977] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5950.256839] LustreError: 393489:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5952.380284] Lustre: 393827:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.145@tcp [ 5953.093081] Lustre: server umount MGS complete [ 5957.075228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5957.254425] LustreError: 394550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.259445] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5957.264406] Lustre: Skipped 1 previous similar message [ 5957.288928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5958.752723] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5961.676039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5961.773450] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5963.260588] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5965.847421] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5967.834179] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5968.479615] Lustre: Permanently reactivating lustre-OST0000 [ 5968.485416] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5968.488356] Lustre: Skipped 2 previous similar messages [ 5993.951182] LustreError: 396329:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5996.851643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.968210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.972021] Lustre: Skipped 1 previous similar message [ 5998.939920] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6001.353682] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6014.433821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.434378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6014.440272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6014.441780] LustreError: Skipped 1 previous similar message [ 6015.378260] Lustre: server umount lustre-OST0000 complete [ 6018.016153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6018.783405] Lustre: server umount lustre-MDT0000 complete [ 6019.551982] LustreError: 394550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.552559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.561402] Lustre: Skipped 2 previous similar messages [ 6020.001164] LustreError: 398171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098851 with bad export cookie 2928479849443377327 [ 6020.005784] LustreError: 398171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6020.005937] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6020.118885] Lustre: server umount lustre-MDT0001 complete [ 6023.564714] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6024.627375] Key type lgssc unregistered [ 6024.763359] LNet: 399220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6024.766729] LNetError: 399220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6024.774285] LNet: Removed LNI 192.168.204.145@tcp [ 6025.061197] Key type .llcrypt unregistered [ 6025.063058] Key type ._llcrypt unregistered [ 6030.939100] Key type ._llcrypt registered [ 6030.940407] Key type .llcrypt registered [ 6030.984820] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 6036.847978] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6037.162854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6037.196735] alg: No test for adler32 (adler32-zlib) [ 6038.041477] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 6038.119358] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6039.695118] Key type lgssc registered [ 6040.045356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6051.619796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6051.625378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6052.710609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6052.720768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6052.758292] Lustre: lustre-MDT0000: new disk, initializing [ 6052.780950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6052.788996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6054.229404] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6058.372464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.417363] Lustre: 402357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6058.449501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6058.451453] Lustre: Skipped 1 previous similar message [ 6058.487874] Lustre: lustre-MDT0001: new disk, initializing [ 6058.526380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6058.539954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6058.543359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6059.903514] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6063.650472] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6065.516338] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6067.886725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6068.011460] Lustre: lustre-OST0000: new disk, initializing [ 6068.013468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6068.016940] Lustre: 404326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6068.053493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.022987] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6073.864120] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6075.454299] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6077.439946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6077.443224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6077.480906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6077.586054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6079.069674] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6079.151231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6082.533090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.533755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6082.538728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6082.539812] LustreError: Skipped 1 previous similar message [ 6082.542903] Lustre: Skipped 1 previous similar message [ 6085.973860] Lustre: server umount lustre-OST0000 complete [ 6087.648674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.649239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.653799] Lustre: Skipped 2 previous similar messages [ 6087.658485] Lustre: Skipped 1 previous similar message [ 6092.768052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6092.770498] Lustre: Skipped 1 previous similar message [ 6093.464923] Lustre: server umount lustre-MDT0000 complete [ 6094.651252] LustreError: 402350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098925 with bad export cookie 10164000890288875398 [ 6094.653635] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6094.655305] LustreError: 402350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6094.776815] Lustre: server umount lustre-MDT0001 complete [ 6097.464728] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6098.009508] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:02:08 (1784098928) [ 6100.141691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.304807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.554789] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6104.165533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.592584] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6107.962132] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6109.712729] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6111.904678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6112.020883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6112.023693] Lustre: Skipped 1 previous similar message [ 6113.895449] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6116.192443] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6116.810048] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6127.811902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6129.631867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6129.634906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.639406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6130.658167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6135.245608] Lustre: server umount lustre-OST0000 complete [ 6140.383973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6140.387648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.393185] Lustre: Skipped 1 previous similar message [ 6140.395672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6140.398734] Lustre: Skipped 2 previous similar messages [ 6142.543148] Lustre: server umount lustre-MDT0000 complete [ 6143.694105] LustreError: 409501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098974 with bad export cookie 10164000890288876266 [ 6143.696474] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6143.697473] LustreError: 409501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6143.810227] Lustre: server umount lustre-MDT0001 complete [ 6147.425570] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6148.431180] Key type lgssc unregistered [ 6148.551372] LNet: 410538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6148.554186] LNetError: 410538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6148.563337] LNet: Removed LNI 192.168.204.145@tcp [ 6148.866035] Key type .llcrypt unregistered [ 6148.867025] Key type ._llcrypt unregistered [ 6156.364647] Key type ._llcrypt registered [ 6156.366061] Key type .llcrypt registered [ 6156.406177] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6156.505338] Key type .llcrypt unregistered [ 6156.506949] Key type ._llcrypt unregistered [ 6163.213035] Key type ._llcrypt registered [ 6163.214061] Key type .llcrypt registered [ 6163.247494] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6163.556632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6163.592420] alg: No test for adler32 (adler32-zlib) [ 6164.444133] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 6164.522820] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6166.095130] Key type lgssc registered [ 6166.417484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6168.846441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6168.854167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.992162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6171.269019] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6174.069512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6174.169570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.986024] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6178.239271] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6179.297661] LustreError: 412724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.304069] LustreError: 412724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6180.321188] LustreError: 412723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6180.327636] LustreError: 412723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6180.327879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6180.467660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.580958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6181.882984] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6190.559967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6190.565257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.571704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6191.077279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6191.078054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6191.080815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.831219] Lustre: server umount lustre-OST0000 complete [ 6196.193590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6196.193861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.194453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.194458] Lustre: Skipped 1 previous similar message [ 6196.203210] Lustre: Skipped 1 previous similar message [ 6201.311892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6201.314017] Lustre: Skipped 2 previous similar messages [ 6202.190587] Lustre: server umount lustre-MDT0000 complete [ 6203.369180] LustreError: 412703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099034 with bad export cookie 1954479675405474691 [ 6203.371783] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.374773] LustreError: 412703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6203.503712] Lustre: server umount lustre-MDT0001 complete [ 6206.943139] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6207.894052] Key type lgssc unregistered [ 6208.019422] LNet: 415871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6208.023390] LNetError: 415871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6208.031503] LNet: Removed LNI 192.168.204.145@tcp [ 6208.311733] Key type .llcrypt unregistered [ 6208.313421] Key type ._llcrypt unregistered [ 6217.075040] Key type ._llcrypt registered [ 6217.076586] Key type .llcrypt registered [ 6217.117359] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6217.526546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6217.553602] alg: No test for adler32 (adler32-zlib) [ 6218.397305] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 6218.496834] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6220.087155] Key type lgssc registered [ 6220.481179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6222.865824] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6222.873313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6238.303331] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6243.423216] LustreError: 417240:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 6270.047180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.863050] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6274.055596] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6276.302389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.533326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.565034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6277.984482] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6281.038346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.192284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6282.492178] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6284.782475] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6286.596277] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6289.377534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.378843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6289.381360] Lustre: Skipped 1 previous similar message [ 6289.384076] Lustre: Skipped 1 previous similar message [ 6294.500291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6294.502575] Lustre: Skipped 1 previous similar message [ 6294.987699] Lustre: server umount lustre-OST0000 complete [ 6296.543910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6296.546704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6296.551375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.616572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.617136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6302.227631] Lustre: server umount lustre-MDT0000 complete [ 6303.371158] LustreError: 417246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099134 with bad export cookie 10508466831344990351 [ 6303.374645] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6303.374944] LustreError: 417246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6303.479029] Lustre: server umount lustre-MDT0001 complete [ 6307.073952] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6308.029722] Key type lgssc unregistered [ 6308.149319] LNet: 420809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6308.151829] LNetError: 420809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6308.162301] LNet: Removed LNI 192.168.204.145@tcp [ 6308.433722] Key type .llcrypt unregistered [ 6308.434851] Key type ._llcrypt unregistered [ 6317.009317] Key type ._llcrypt registered [ 6317.010823] Key type .llcrypt registered [ 6317.046674] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6317.305791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6317.311073] alg: No test for adler32 (adler32-zlib) [ 6318.150462] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 6318.238369] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6319.823129] Key type lgssc registered [ 6320.160114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6322.509995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6322.516958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.658317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.976401] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6327.732825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.851572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6329.133601] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6331.462381] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6333.290671] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6335.532438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6335.644145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6337.459784] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6339.689858] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6348.963982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6353.376343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6353.378734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6353.383608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6354.400309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6354.401635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.402787] Lustre: Skipped 1 previous similar message [ 6354.406958] Lustre: Skipped 1 previous similar message [ 6355.588726] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 6357.529756] Lustre: server umount lustre-MDT0000 complete [ 6358.285692] LustreError: 422183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.45@tcp arrived at 1784099189 with bad export cookie 13675915235908234272 [ 6358.290080] LustreError: 422183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6358.939355] LustreError: 422935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099190 with bad export cookie 13675915235908233621 [ 6358.943430] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6358.944700] LustreError: 422935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6359.519855] LustreError: 422201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6359.520525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6359.524956] LustreError: 422201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6359.530551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6363.551876] LustreError: 422200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.552234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6363.558535] LustreError: 422200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6365.069664] Lustre: server umount lustre-OST0000 complete [ 6367.254832] Lustre: server umount lustre-MDT0001 complete [ 6370.914706] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6371.934202] Key type lgssc unregistered [ 6372.069390] LNet: 425901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6372.072482] LNetError: 425901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6372.082276] LNet: Removed LNI 192.168.204.145@tcp [ 6372.385139] Key type .llcrypt unregistered [ 6372.386231] Key type ._llcrypt unregistered [ 6381.347815] Key type ._llcrypt registered [ 6381.349301] Key type .llcrypt registered [ 6381.391331] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6381.789533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6381.842734] alg: No test for adler32 (adler32-zlib) [ 6382.691365] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 6382.774612] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6384.351128] Key type lgssc registered [ 6384.695767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6387.051526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6387.058851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6388.200474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6389.467593] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6392.056054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.165722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6393.439425] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6395.831712] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6397.536228] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6399.624164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.731817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.435773] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6403.655906] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6413.030064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6417.887767] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6417.890022] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6417.895094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6418.400441] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6418.401793] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.521324] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6423.523054] Lustre: Skipped 3 previous similar messages [ 6428.640673] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6428.643247] Lustre: Skipped 2 previous similar messages [ 6429.151102] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6429.198432] Lustre: server umount lustre-MDT0001 complete [ 6429.309219] LustreError: 427298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.315946] LustreError: 427298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6433.761995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6433.762018] LustreError: 428031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.762368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6433.776447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6436.824136] Lustre: server umount lustre-OST0000 complete [ 6438.123849] Lustre: server umount lustre-MDT0000 complete [ 6442.692724] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6443.752773] Key type lgssc unregistered [ 6443.885413] LNet: 431009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6443.887693] LNetError: 431009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6443.895398] LNet: Removed LNI 192.168.204.145@tcp [ 6444.204665] Key type .llcrypt unregistered [ 6444.206098] Key type ._llcrypt unregistered [ 6452.195137] Key type ._llcrypt registered [ 6452.196821] Key type .llcrypt registered [ 6452.231162] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 6458.165199] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6458.461495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6458.506026] alg: No test for adler32 (adler32-zlib) [ 6459.362399] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 6459.458362] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6461.039183] Key type lgssc registered [ 6461.443567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6473.257378] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6478.389932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6478.396561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.474649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6479.485881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6479.526376] Lustre: lustre-MDT0000: new disk, initializing [ 6479.547246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.552993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6480.806598] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6486.305204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6486.427399] Lustre: lustre-OST0000: new disk, initializing [ 6486.428910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6486.430948] Lustre: Skipped 1 previous similar message [ 6486.432498] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6486.434356] Lustre: 436014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6486.457780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.286548] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6491.627952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6491.630444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6491.637503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6491.805383] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6497.112059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.156921] Lustre: 434993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6497.184940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6497.221453] Lustre: lustre-MDT0001: new disk, initializing [ 6497.260795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6497.272101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6497.275385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6498.549883] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6501.949913] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6502.024787] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6503.305962] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6504.394624] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6508.513202] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6508.513719] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6508.514695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6508.517831] Lustre: Skipped 1 previous similar message [ 6510.211435] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 6510.213374] Lustre: Skipped 1 previous similar message [ 6513.634549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6513.636329] Lustre: Skipped 1 previous similar message [ 6518.753085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6518.755199] Lustre: Skipped 2 previous similar messages [ 6520.287090] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6520.349623] Lustre: server umount lustre-MDT0001 complete [ 6528.993151] LustreError: 435002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6528.997109] LustreError: 435002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6534.112240] LustreError: 437351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.112588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6534.121134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.707753] Lustre: server umount lustre-OST0000 complete [ 6539.232740] LustreError: 435006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.233286] Lustre: server umount lustre-MDT0000 complete [ 6550.765298] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6551.765548] Key type lgssc unregistered [ 6551.890388] LNet: 439767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6551.892646] LNetError: 439767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6551.904298] LNet: Removed LNI 192.168.204.145@tcp [ 6552.201915] Key type .llcrypt unregistered [ 6552.203022] Key type ._llcrypt unregistered [ 6564.027880] Key type ._llcrypt registered [ 6564.028860] Key type .llcrypt registered [ 6564.062959] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6564.163257] Key type .llcrypt unregistered [ 6564.164407] Key type ._llcrypt unregistered [ 6570.803150] Key type ._llcrypt registered [ 6570.804571] Key type .llcrypt registered [ 6570.844450] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6571.176836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6571.219107] alg: No test for adler32 (adler32-zlib) [ 6572.077743] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 6572.173909] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6573.751216] Key type lgssc registered [ 6574.087932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6582.566582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6582.571912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6583.656129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6583.668450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6583.709283] Lustre: lustre-MDT0000: new disk, initializing [ 6583.732088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.739226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6585.081736] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6589.174684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6589.222536] Lustre: 443115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6589.254263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6589.256737] Lustre: Skipped 1 previous similar message [ 6589.277168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6589.297900] Lustre: lustre-MDT0001: new disk, initializing [ 6589.334289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6589.347733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6589.352255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6590.614201] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6594.302398] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6596.165113] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6598.419026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6598.525530] Lustre: lustre-OST0000: new disk, initializing [ 6598.527865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6598.531351] Lustre: 445082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6598.561504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6600.477220] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6604.193048] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6607.352389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6607.357332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6607.385037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6612.449798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.450880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6612.456206] Lustre: Skipped 2 previous similar messages [ 6612.458960] Lustre: Skipped 1 previous similar message [ 6617.568448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6617.570774] Lustre: Skipped 3 previous similar messages [ 6617.818261] Lustre: server umount lustre-MDT0000 complete [ 6619.031595] LustreError: 443108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099450 with bad export cookie 3707456048196372087 [ 6619.034606] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6619.036544] LustreError: 443108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6619.151798] Lustre: server umount lustre-MDT0001 complete [ 6630.610556] Lustre: server umount lustre-OST0000 complete [ 6633.836806] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:11:04 (1784099464) [ 6636.371330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6636.547722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6637.961448] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6640.650986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.758370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6642.071334] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6644.487487] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6646.289342] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6661.473577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.548766] Lustre: Found index 0 for lustre-OST0000, updating log [ 6661.618104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6663.487293] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6665.957267] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6682.085579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6691.296060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6691.298365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.302295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6694.093545] Lustre: server umount lustre-OST0000 complete [ 6706.639109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.640075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.643561] Lustre: Skipped 2 previous similar messages [ 6706.645883] Lustre: Skipped 3 previous similar messages [ 6712.785902] Lustre: server umount lustre-MDT0000 complete [ 6713.926192] LustreError: 448949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099545 with bad export cookie 3707456048196373480 [ 6713.928928] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6713.930011] LustreError: 448949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6714.037573] Lustre: server umount lustre-MDT0001 complete [ 6716.645373] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:12:27 (1784099547) [ 6733.983337] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [ 7296.062776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.227272] LustreError: 453529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7296.234849] LustreError: 453529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7296.252520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.666330] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7300.430647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7300.555847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7301.897401] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7304.159432] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7305.876975] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7321.185861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.272163] Lustre: Found index 0 for lustre-OST0000, updating log [ 7321.382088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7323.245131] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7323.430833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7325.527528] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7327.265943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7327.273517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7327.275190] Lustre: Skipped 3 previous similar messages [ 7328.289469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7328.293444] Lustre: Skipped 1 previous similar message [ 7332.335448] Lustre: server umount lustre-OST0000 complete [ 7334.756335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.758846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7334.766205] Lustre: Skipped 1 previous similar message [ 7334.773634] Lustre: Skipped 1 previous similar message [ 7338.850160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7338.855017] Lustre: Skipped 7 previous similar messages [ 7347.040323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7347.042883] Lustre: Skipped 15 previous similar messages [ 7348.191082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7348.259532] Lustre: server umount lustre-MDT0000 complete [ 7348.383163] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [ 7349.088862] LustreError: 453529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7349.096222] LustreError: 453529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7349.790181] LustreError: 453511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100180 with bad export cookie 3707456048196374551 [ 7349.797410] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7349.798702] LustreError: 453511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7349.934809] Lustre: server umount lustre-MDT0001 complete [ 7367.341127] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7370.797415] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7383.156649] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:23:33 (1784100213) [ 7389.577826] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7390.517053] Key type lgssc unregistered [ 7399.839342] LNet: 460183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7399.842753] LNetError: 460183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7452.069277] LNet: Removed LNI 192.168.204.145@tcp [ 7452.360536] Key type .llcrypt unregistered [ 7452.361870] Key type ._llcrypt unregistered [ 7459.779394] Key type ._llcrypt registered [ 7459.780469] Key type .llcrypt registered [ 7459.820790] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7460.246953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7460.257195] alg: No test for adler32 (adler32-zlib) [ 7461.100754] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 7461.203506] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7462.783188] Key type lgssc registered [ 7463.225979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7464.906448] LNet: Added LNI 192.168.204.145@tcp100 [8/256/0/180] [ 7465.227829] LNet: Added LNI 192.168.204.145@tcp101 [8/256/0/180] [ 7465.578859] LNet: Added LNI 192.168.204.145@tcp102 [8/256/0/180] [ 7466.037050] LNet: Added LNI 192.168.204.145@tcp103 [8/256/0/180] [ 7466.480212] LNet: Added LNI 192.168.204.145@tcp104 [8/256/0/180] [ 7466.822881] LNet: Added LNI 192.168.204.145@tcp105 [8/256/0/180] [ 7467.095874] LNet: Added LNI 192.168.204.145@tcp106 [8/256/0/180] [ 7467.375476] LNet: Added LNI 192.168.204.145@tcp107 [8/256/0/180] [ 7467.669468] LNet: Added LNI 192.168.204.145@tcp108 [8/256/0/180] [ 7467.982986] LNet: Added LNI 192.168.204.145@tcp109 [8/256/0/180] [ 7468.335744] LNet: Added LNI 192.168.204.145@tcp110 [8/256/0/180] [ 7468.736794] LNet: Added LNI 192.168.204.145@tcp111 [8/256/0/180] [ 7469.144751] LNet: Added LNI 192.168.204.145@tcp112 [8/256/0/180] [ 7469.583722] LNet: Added LNI 192.168.204.145@tcp113 [8/256/0/180] [ 7470.024554] LNet: Added LNI 192.168.204.145@tcp114 [8/256/0/180] [ 7470.388231] LNet: Added LNI 192.168.204.145@tcp115 [8/256/0/180] [ 7470.826783] LNet: Added LNI 192.168.204.145@tcp116 [8/256/0/180] [ 7471.265564] LNet: Added LNI 192.168.204.145@tcp117 [8/256/0/180] [ 7471.648357] LNet: Added LNI 192.168.204.145@tcp118 [8/256/0/180] [ 7472.005878] LNet: Added LNI 192.168.204.145@tcp119 [8/256/0/180] [ 7472.408860] LNet: Added LNI 192.168.204.145@tcp120 [8/256/0/180] [ 7472.763541] LNet: Added LNI 192.168.204.145@tcp121 [8/256/0/180] [ 7473.148145] LNet: Added LNI 192.168.204.145@tcp122 [8/256/0/180] [ 7473.512171] LNet: Added LNI 192.168.204.145@tcp123 [8/256/0/180] [ 7473.938724] LNet: Added LNI 192.168.204.145@tcp124 [8/256/0/180] [ 7474.392470] LNet: Added LNI 192.168.204.145@tcp125 [8/256/0/180] [ 7474.839533] LNet: Added LNI 192.168.204.145@tcp126 [8/256/0/180] [ 7475.213296] LNet: Added LNI 192.168.204.145@tcp127 [8/256/0/180] [ 7475.654282] LNet: Added LNI 192.168.204.145@tcp128 [8/256/0/180] [ 7476.039663] LNet: Added LNI 192.168.204.145@tcp129 [8/256/0/180] [ 7476.344907] LNet: Added LNI 192.168.204.145@tcp130 [8/256/0/180] [ 7476.698290] LNet: Added LNI 192.168.204.145@tcp131 [8/256/0/180] [ 7477.058025] LNet: Added LNI 192.168.204.145@tcp132 [8/256/0/180] [ 7477.461581] LNet: Added LNI 192.168.204.145@tcp133 [8/256/0/180] [ 7477.848652] LNet: Added LNI 192.168.204.145@tcp134 [8/256/0/180] [ 7478.265561] LNet: Added LNI 192.168.204.145@tcp135 [8/256/0/180] [ 7481.058878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7481.066112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.110321] Lustre: 463180:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [ 7482.136490] LNetError: 463202:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [ 7482.142836] LNetError: Protocol error connecting to 192.168.204.145@tcp119 at host 192.168.204.145:988: is it running a compatible version of Lustre? [ 7482.149255] LNetError: Skipped 4 previous similar messages [ 7482.184555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7482.194920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7482.231389] Lustre: lustre-MDT0000: new disk, initializing [ 7482.256072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.264126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7482.720088] LNet: 2 local NIs in recovery (showing 2): 192.168.204.145@tcp123, 192.168.204.145@tcp119 [ 7483.167927] LNetError: 463210:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [ 7483.170887] LNetError: 463210:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 7483.173171] LNetError: Protocol error connecting to 192.168.204.145@tcp106 at host 192.168.204.145:988: is it running a compatible version of Lustre? [ 7483.488390] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7485.792303] LNetError: 463200:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [ 7485.795530] LNetError: 463200:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 7485.798047] LNetError: Protocol error connecting to 192.168.204.145@tcp123 at host 192.168.204.145:988: is it running a compatible version of Lustre? [ 7485.801339] LNetError: Skipped 2 previous similar messages [ 7487.423277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.471275] Lustre: 463198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7487.494948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7487.496973] Lustre: Skipped 1 previous similar message [ 7487.531705] Lustre: lustre-MDT0001: new disk, initializing [ 7487.544597] Lustre: 463999:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [ 7487.548837] Lustre: 463999:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7487.586575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7487.598346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7487.601501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7488.791407] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7489.887991] LNetError: 460643:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [ 7489.890874] LNetError: Protocol error connecting to 192.168.204.145@tcp123 at host 192.168.204.145:988: is it running a compatible version of Lustre? [ 7492.231784] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7493.852630] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7495.667262] Lustre: 465105:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [ 7495.670497] Lustre: 465105:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7495.907758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.014671] Lustre: lustre-OST0000: new disk, initializing [ 7496.017240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7496.020695] Lustre: 465172:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7496.068841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.055944] LNetError: 460644:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [ 7497.058768] LNetError: Protocol error connecting to 192.168.204.145@tcp123 at host 192.168.204.145:988: is it running a compatible version of Lustre? [ 7497.841569] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7500.784107] Lustre: 465833:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [ 7500.787129] Lustre: 465833:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [ 7500.803515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7500.807077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7500.851357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7501.247076] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7505.888896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7505.891310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.895230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7505.897062] Lustre: Skipped 1 previous similar message [ 7509.710208] Lustre: server umount lustre-OST0000 complete [ 7511.008878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7511.009626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7511.013193] Lustre: Skipped 2 previous similar messages [ 7511.018326] Lustre: Skipped 1 previous similar message [ 7513.441202] LNetError: 460642:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [ 7513.444304] LNetError: Protocol error connecting to 192.168.204.145@tcp123 at host 192.168.204.145:988: is it running a compatible version of Lustre? [ 7516.128202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.130134] Lustre: Skipped 1 previous similar message [ 7521.248166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7521.250828] Lustre: Skipped 1 previous similar message [ 7525.343124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7525.406912] Lustre: server umount lustre-MDT0000 complete [ 7526.618623] LustreError: 463189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100357 with bad export cookie 14591964091816761695 [ 7526.622488] LustreError: 463189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7526.623184] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7526.740453] Lustre: server umount lustre-MDT0001 complete [ 7529.823923] LNetError: 460642:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [ 7529.826635] LNetError: 460642:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 7529.828739] LNetError: Protocol error connecting to 192.168.204.145@tcp124 at host 192.168.204.145:988: is it running a compatible version of Lustre? [ 7529.832420] LNetError: Skipped 7 previous similar messages [ 7530.388143] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7531.357997] Key type lgssc unregistered [ 7531.508635] LNet: 467084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7531.512233] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7531.514957] LNet: Removed LNI 192.168.204.145@tcp [ 7532.575086] LNet: Removed LNI 192.168.204.145@tcp100 [ 7532.576562] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7532.579073] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7532.582468] LNet: Removed LNI 192.168.204.145@tcp101 [ 7533.599100] LNet: Removed LNI 192.168.204.145@tcp102 [ 7533.600507] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7533.603302] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7533.606197] LNet: Removed LNI 192.168.204.145@tcp103 [ 7534.623090] LNet: Removed LNI 192.168.204.145@tcp104 [ 7535.647151] LNet: Removed LNI 192.168.204.145@tcp105 [ 7535.649474] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7535.653330] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7535.657472] LNet: Removed LNI 192.168.204.145@tcp106 [ 7536.671158] LNet: Removed LNI 192.168.204.145@tcp107 [ 7537.695157] LNet: Removed LNI 192.168.204.145@tcp108 [ 7538.719147] LNet: Removed LNI 192.168.204.145@tcp109 [ 7539.743151] LNet: Removed LNI 192.168.204.145@tcp110 [ 7539.745186] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7539.747319] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7539.750610] LNet: Removed LNI 192.168.204.145@tcp111 [ 7540.767135] LNet: Removed LNI 192.168.204.145@tcp112 [ 7541.791092] LNet: Removed LNI 192.168.204.145@tcp113 [ 7542.815100] LNet: Removed LNI 192.168.204.145@tcp114 [ 7543.839129] LNet: Removed LNI 192.168.204.145@tcp115 [ 7544.863140] LNet: Removed LNI 192.168.204.145@tcp116 [ 7545.887136] LNet: Removed LNI 192.168.204.145@tcp117 [ 7546.911141] LNet: Removed LNI 192.168.204.145@tcp118 [ 7547.935100] LNet: Removed LNI 192.168.204.145@tcp119 [ 7547.936469] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.938749] LNetError: 467084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7547.940969] LNet: Removed LNI 192.168.204.145@tcp120 [ 7548.959140] LNet: Removed LNI 192.168.204.145@tcp121 [ 7549.983177] LNet: Removed LNI 192.168.204.145@tcp122 [ 7549.985419] LNet: Removed LNI 192.168.204.145@tcp123 [ 7551.007142] LNet: Removed LNI 192.168.204.145@tcp124 [ 7551.009084] LNet: Removed LNI 192.168.204.145@tcp125 [ 7551.010860] LNet: Removed LNI 192.168.204.145@tcp126 [ 7551.012524] LNet: Removed LNI 192.168.204.145@tcp127 [ 7551.014133] LNet: Removed LNI 192.168.204.145@tcp128 [ 7551.015996] LNet: Removed LNI 192.168.204.145@tcp129 [ 7551.017708] LNet: Removed LNI 192.168.204.145@tcp130 [ 7551.019243] LNet: Removed LNI 192.168.204.145@tcp131 [ 7551.020992] LNet: Removed LNI 192.168.204.145@tcp132 [ 7551.022768] LNet: Removed LNI 192.168.204.145@tcp133 [ 7551.024367] LNet: Removed LNI 192.168.204.145@tcp134 [ 7551.031421] LNet: Removed LNI 192.168.204.145@tcp135 [ 7551.307665] Key type .llcrypt unregistered [ 7551.308819] Key type ._llcrypt unregistered [ 7558.817368] Key type ._llcrypt registered [ 7558.818479] Key type .llcrypt registered [ 7558.852566] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7564.965875] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7565.296940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7565.343626] alg: No test for adler32 (adler32-zlib) [ 7566.206751] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 7566.300564] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7567.887201] Key type lgssc registered [ 7568.426682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7579.478762] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:26:49 (1784100409) [ 7585.929582] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7586.916241] Key type lgssc unregistered [ 7587.043401] LNet: 471141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7587.046293] LNetError: 471141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7587.058384] LNet: Removed LNI 192.168.204.145@tcp [ 7587.416392] Key type .llcrypt unregistered [ 7587.418626] Key type ._llcrypt unregistered [ 7594.280466] Key type ._llcrypt registered [ 7594.281423] Key type .llcrypt registered [ 7594.324498] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7594.835473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7594.843696] alg: No test for adler32 (adler32-zlib) [ 7595.731114] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 7595.834471] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7597.423145] Key type lgssc registered [ 7597.898434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7600.881519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7600.889741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.980132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7601.991978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7602.040896] Lustre: lustre-MDT0000: new disk, initializing [ 7602.065415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7602.071968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7603.457869] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7607.843532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.891950] Lustre: 472330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7607.922863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7607.926348] Lustre: Skipped 1 previous similar message [ 7607.971064] Lustre: lustre-MDT0001: new disk, initializing [ 7608.025140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7608.041336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7608.046451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7609.744233] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7613.700221] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7615.482869] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7617.677487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7617.770401] Lustre: lustre-OST0000: new disk, initializing [ 7617.771783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7617.774238] Lustre: 474301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7617.800942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7618.749941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7618.754220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7618.782840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7619.722756] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7623.428238] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7624.707608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lnet_if_list [ 7626.402308] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7626.413528] LNet: Added LNI 192.168.204.145@tcp10 [8/256/0/180] [ 7628.242888] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7628.256733] LNet: Added LNI 192.168.204.145@tcp11 [8/256/0/180] [ 7629.661209] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7629.673524] LNet: Added LNI 192.168.204.145@tcp12 [8/256/0/180] [ 7631.116020] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7631.130174] LNet: Added LNI 192.168.204.145@tcp13 [8/256/0/180] [ 7632.595495] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7632.607841] LNet: Added LNI 192.168.204.145@tcp14 [8/256/0/180] [ 7634.114705] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7634.125310] LNet: Added LNI 192.168.204.145@tcp15 [8/256/0/180] [ 7635.544906] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7635.556361] LNet: Added LNI 192.168.204.145@tcp16 [8/256/0/180] [ 7637.086264] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7637.097404] LNet: Added LNI 192.168.204.145@tcp17 [8/256/0/180] [ 7638.562406] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7638.573143] LNet: Added LNI 192.168.204.145@tcp18 [8/256/0/180] [ 7640.099153] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7640.114026] LNet: Added LNI 192.168.204.145@tcp19 [8/256/0/180] [ 7641.710586] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7641.723265] LNet: Added LNI 192.168.204.145@tcp20 [8/256/0/180] [ 7643.324809] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7643.335984] LNet: Added LNI 192.168.204.145@tcp21 [8/256/0/180] [ 7644.987402] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7645.000673] LNet: Added LNI 192.168.204.145@tcp22 [8/256/0/180] [ 7646.657911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7646.670752] LNet: Added LNI 192.168.204.145@tcp23 [8/256/0/180] [ 7648.326957] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7648.338705] LNet: Added LNI 192.168.204.145@tcp24 [8/256/0/180] [ 7650.022940] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7650.034905] LNet: Added LNI 192.168.204.145@tcp25 [8/256/0/180] [ 7651.775647] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7651.785253] LNet: Added LNI 192.168.204.145@tcp26 [8/256/0/180] [ 7653.535661] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7653.546332] LNet: Added LNI 192.168.204.145@tcp27 [8/256/0/180] [ 7655.457934] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7655.467510] LNet: Added LNI 192.168.204.145@tcp28 [8/256/0/180] [ 7657.446133] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7657.459776] LNet: Added LNI 192.168.204.145@tcp29 [8/256/0/180] [ 7659.535282] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7659.546848] LNet: Added LNI 192.168.204.145@tcp30 [8/256/0/180] [ 7661.519788] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7661.529753] LNet: Added LNI 192.168.204.145@tcp31 [8/256/0/180] [ 7663.555760] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7663.567290] LNet: Added LNI 192.168.204.145@tcp32 [8/256/0/180] [ 7665.604517] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7665.617847] LNet: Added LNI 192.168.204.145@tcp33 [8/256/0/180] [ 7667.851952] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7667.862661] LNet: Added LNI 192.168.204.145@tcp34 [8/256/0/180] [ 7670.074553] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7670.086749] LNet: Added LNI 192.168.204.145@tcp35 [8/256/0/180] [ 7672.395114] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7672.405586] LNet: Added LNI 192.168.204.145@tcp36 [8/256/0/180] [ 7674.568514] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7674.577494] LNet: Added LNI 192.168.204.145@tcp37 [8/256/0/180] [ 7676.802986] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7676.812890] LNet: Added LNI 192.168.204.145@tcp38 [8/256/0/180] [ 7679.008569] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7679.019434] LNet: Added LNI 192.168.204.145@tcp39 [8/256/0/180] [ 7681.224907] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7681.235438] LNet: Added LNI 192.168.204.145@tcp40 [8/256/0/180] [ 7683.417585] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7683.426734] LNet: Added LNI 192.168.204.145@tcp41 [8/256/0/180] [ 7685.638651] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7685.648607] LNet: Added LNI 192.168.204.145@tcp42 [8/256/0/180] [ 7687.872246] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7687.882279] LNet: Added LNI 192.168.204.145@tcp43 [8/256/0/180] [ 7690.216122] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7690.228473] LNet: Added LNI 192.168.204.145@tcp44 [8/256/0/180] [ 7692.525545] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7692.535546] LNet: Added LNI 192.168.204.145@tcp45 [8/256/0/180] [ 7694.815392] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7694.825288] LNet: Added LNI 192.168.204.145@tcp46 [8/256/0/180] [ 7697.216552] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7697.228718] LNet: Added LNI 192.168.204.145@tcp47 [8/256/0/180] [ 7699.960705] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7699.977717] LNet: Added LNI 192.168.204.145@tcp48 [8/256/0/180] [ 7703.048494] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7703.066179] LNet: Added LNI 192.168.204.145@tcp49 [8/256/0/180] [ 7705.751216] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7705.765361] LNet: Added LNI 192.168.204.145@tcp50 [8/256/0/180] [ 7715.808034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7715.812322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7715.817598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7716.321316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7716.324082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.327762] Lustre: Skipped 1 previous similar message [ 7719.571465] Lustre: server umount lustre-MDT0000 complete [ 7720.755032] LustreError: 472325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100551 with bad export cookie 10590769400116504589 [ 7720.757501] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7720.759124] LustreError: 472325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7720.864242] Lustre: server umount lustre-MDT0001 complete [ 7728.261827] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7731.233298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.823112] Lustre: 471607:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784100552/real 1784100552] req@00000000ea4037a8 x1870764889423616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784100568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7737.836085] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.831147] Lustre: 471604:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784100556/real 1784100556] req@00000000128293ac x1870764889423744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784100572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7741.855129] Lustre: 471604:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784100557/real 1784100557] req@00000000dbf2626f x1870764889423872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784100573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7744.991407] LustreError: 471603:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b474dc5e x1870764889424768/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7745.116069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7746.015155] Lustre: 471607:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784100561/real 1784100561] req@00000000e4cd94f5 x1870764889424128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784100577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7746.505521] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7749.300463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.427202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7750.782569] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7751.455117] Lustre: 471607:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784100566/real 1784100566] req@0000000046ba6001 x1870764889424256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784100582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7753.081864] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7754.724559] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7754.892089] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7756.773560] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7764.960613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7764.963359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.967241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7764.968943] Lustre: Skipped 2 previous similar messages [ 7767.009181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7767.011049] Lustre: Skipped 2 previous similar messages [ 7768.343833] Lustre: server umount lustre-OST0000 complete [ 7770.079810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7770.081911] LustreError: Skipped 1 previous similar message [ 7770.083039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.086242] Lustre: Skipped 1 previous similar message [ 7772.128223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7772.130801] Lustre: Skipped 2 previous similar messages [ 7775.711559] Lustre: server umount lustre-MDT0000 complete [ 7777.248159] LustreError: 498491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7777.252966] LustreError: 498491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7777.335063] LustreError: 472325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100608 with bad export cookie 10590769400116506654 [ 7777.338527] LustreError: 472325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7777.338727] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7777.433168] Lustre: server umount lustre-MDT0001 complete [ 7781.182449] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7782.305937] Key type lgssc unregistered [ 7782.455455] LNet: 501265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7782.458835] LNetError: 501265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.461235] LNet: Removed LNI 192.168.204.145@tcp [ 7782.462316] LNet: Removed LNI 192.168.204.145@tcp10 [ 7782.463494] LNet: Removed LNI 192.168.204.145@tcp11 [ 7782.464555] LNet: Removed LNI 192.168.204.145@tcp12 [ 7782.465770] LNet: Removed LNI 192.168.204.145@tcp13 [ 7782.466876] LNet: Removed LNI 192.168.204.145@tcp14 [ 7782.468117] LNet: Removed LNI 192.168.204.145@tcp15 [ 7782.469428] LNet: Removed LNI 192.168.204.145@tcp16 [ 7782.470566] LNet: Removed LNI 192.168.204.145@tcp17 [ 7782.472025] LNet: Removed LNI 192.168.204.145@tcp18 [ 7782.473286] LNet: Removed LNI 192.168.204.145@tcp19 [ 7782.474384] LNet: Removed LNI 192.168.204.145@tcp20 [ 7782.475572] LNet: Removed LNI 192.168.204.145@tcp21 [ 7782.476648] LNet: Removed LNI 192.168.204.145@tcp22 [ 7782.477777] LNet: Removed LNI 192.168.204.145@tcp23 [ 7782.478909] LNet: Removed LNI 192.168.204.145@tcp24 [ 7782.479960] LNet: Removed LNI 192.168.204.145@tcp25 [ 7782.481284] LNet: Removed LNI 192.168.204.145@tcp26 [ 7782.482500] LNet: Removed LNI 192.168.204.145@tcp27 [ 7782.483794] LNet: Removed LNI 192.168.204.145@tcp28 [ 7782.484956] LNet: Removed LNI 192.168.204.145@tcp29 [ 7782.486078] LNet: Removed LNI 192.168.204.145@tcp30 [ 7782.487261] LNet: Removed LNI 192.168.204.145@tcp31 [ 7782.488371] LNet: Removed LNI 192.168.204.145@tcp32 [ 7782.489593] LNet: Removed LNI 192.168.204.145@tcp33 [ 7782.490816] LNet: Removed LNI 192.168.204.145@tcp34 [ 7782.492109] LNet: Removed LNI 192.168.204.145@tcp35 [ 7782.493320] LNet: Removed LNI 192.168.204.145@tcp36 [ 7782.494555] LNet: Removed LNI 192.168.204.145@tcp37 [ 7782.495737] LNet: Removed LNI 192.168.204.145@tcp38 [ 7782.496917] LNet: Removed LNI 192.168.204.145@tcp39 [ 7782.498113] LNet: Removed LNI 192.168.204.145@tcp40 [ 7782.499312] LNet: Removed LNI 192.168.204.145@tcp41 [ 7782.500439] LNet: Removed LNI 192.168.204.145@tcp42 [ 7782.501589] LNet: Removed LNI 192.168.204.145@tcp43 [ 7782.502737] LNet: Removed LNI 192.168.204.145@tcp44 [ 7782.503799] LNet: Removed LNI 192.168.204.145@tcp45 [ 7782.504848] LNet: Removed LNI 192.168.204.145@tcp46 [ 7782.505901] LNet: Removed LNI 192.168.204.145@tcp47 [ 7782.507077] LNet: Removed LNI 192.168.204.145@tcp48 [ 7782.508070] LNet: Removed LNI 192.168.204.145@tcp49 [ 7782.515338] LNet: Removed LNI 192.168.204.145@tcp50 [ 7782.801880] Key type .llcrypt unregistered [ 7782.803374] Key type ._llcrypt unregistered [ 7789.220202] Key type ._llcrypt registered [ 7789.221233] Key type .llcrypt registered [ 7789.258237] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7796.061731] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7796.389317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7796.421556] alg: No test for adler32 (adler32-zlib) [ 7797.265928] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 7797.348880] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7798.919130] Key type lgssc registered [ 7799.243813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7810.301461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7810.310589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.399984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7811.410133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7811.456553] Lustre: lustre-MDT0000: new disk, initializing [ 7811.481239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7811.487361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7812.863704] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7817.048424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.097499] Lustre: 504431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7817.122726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7817.124870] Lustre: Skipped 1 previous similar message [ 7817.166884] Lustre: lustre-MDT0001: new disk, initializing [ 7817.205885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.220356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7817.223918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7818.655445] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7822.590660] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7824.512270] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7826.981330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7827.055541] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7827.083714] Lustre: lustre-OST0000: new disk, initializing [ 7827.085505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7827.088469] Lustre: 506446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7827.116792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7829.097007] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7832.795914] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7833.080941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7833.083801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7833.117723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7838.179722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.179730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7838.180539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7838.183846] Lustre: Skipped 1 previous similar message [ 7841.179715] Lustre: server umount lustre-OST0000 complete [ 7842.784692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7842.791064] LustreError: Skipped 1 previous similar message [ 7842.794711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.806489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7842.811624] Lustre: Skipped 1 previous similar message [ 7848.416298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7848.418786] Lustre: Skipped 3 previous similar messages [ 7848.600957] Lustre: server umount lustre-MDT0000 complete [ 7849.823211] LustreError: 504424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100680 with bad export cookie 9230750323535686700 [ 7849.826811] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7849.827417] LustreError: 504424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7849.946050] Lustre: server umount lustre-MDT0001 complete [ 7853.712966] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7854.677954] Key type lgssc unregistered [ 7854.801319] LNet: 508351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7854.804536] LNetError: 508351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7854.813297] LNet: Removed LNI 192.168.204.145@tcp [ 7855.119802] Key type .llcrypt unregistered [ 7855.121270] Key type ._llcrypt unregistered [ 7862.673129] Key type ._llcrypt registered [ 7862.674217] Key type .llcrypt registered [ 7862.706226] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7863.096631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7863.103212] alg: No test for adler32 (adler32-zlib) [ 7863.953700] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 7864.050190] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7865.631137] Key type lgssc registered [ 7865.990295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7869.054510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7869.061994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7870.213794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7871.585025] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7874.630373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.797651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7876.394789] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7879.091091] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7881.105551] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7883.888660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7884.003453] LustreError: 511414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.012766] LustreError: 511414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7884.093957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7886.053472] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7888.329169] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7902.306180] loop0: detected capacity change from 0 to 204800000 [ 7902.390535] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7902.394803] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7902.398596] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7904.558859] loop0: detected capacity change from 0 to 204800000 [ 7904.623863] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7904.627569] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7904.632169] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7906.313528] loop0: detected capacity change from 0 to 204800000 [ 7908.117647] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7908.172051] Lustre: fs15246-OST0000: new disk, initializing [ 7908.174091] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7908.177704] Lustre: 513210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7908.196929] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.092655] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7913.910619] loop1: detected capacity change from 0 to 204800000 [ 7915.851573] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7915.900689] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7915.929614] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7915.972051] Lustre: fs15246-MDT0000: new disk, initializing [ 7916.030019] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7916.040415] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7916.091665] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7917.151925] Lustre: 514332:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d3d69479 x1870765170392832/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784100764 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7917.404368] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7922.208962] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7922.214044] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7922.219364] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7926.757318] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.762405] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7931.283050] Lustre: server umount fs15246-MDT0000 complete [ 7931.876288] LustreError: 510943:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7934.441578] Lustre: server umount fs15246-OST0000 complete [ 7940.575731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7940.577878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.581682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7941.602305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7941.608270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7941.618564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7947.235115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7947.240027] Lustre: Skipped 3 previous similar messages [ 7952.351135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7952.355938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7952.357886] Lustre: Skipped 1 previous similar message [ 7952.408543] Lustre: server umount lustre-OST0000 complete [ 7956.959759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7956.961992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.768898] Lustre: server umount lustre-MDT0000 complete [ 7960.961043] LustreError: 509579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100792 with bad export cookie 14332454825091603319 [ 7960.964925] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.966316] LustreError: 509579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7961.069637] Lustre: server umount lustre-MDT0001 complete [ 7964.332259] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7965.316777] Key type lgssc unregistered [ 7965.445311] LNet: 517426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7965.447985] LNetError: 517426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7965.455306] LNet: Removed LNI 192.168.204.145@tcp [ 7965.748675] Key type .llcrypt unregistered [ 7965.750396] Key type ._llcrypt unregistered [ 7972.737193] Key type ._llcrypt registered [ 7972.738378] Key type .llcrypt registered [ 7972.770424] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7973.127964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7973.154293] alg: No test for adler32 (adler32-zlib) [ 7974.010959] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 7974.095633] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7975.679104] Key type lgssc registered [ 7976.028800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7988.168457] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7991.128207] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8002.087431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8002.092325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.171832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8003.181159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8003.216715] Lustre: lustre-MDT0000: new disk, initializing [ 8003.238274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.244143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8004.501503] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8008.465399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8008.521844] Lustre: 521861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8008.549789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8008.551847] Lustre: Skipped 1 previous similar message [ 8008.591135] Lustre: lustre-MDT0001: new disk, initializing [ 8008.627414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8008.644193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8008.649557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8010.073378] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8014.251278] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8016.428412] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8018.684559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8018.782562] Lustre: lustre-OST0000: new disk, initializing [ 8018.784365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8018.786833] Lustre: 523826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8018.815753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.960268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8019.963802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8019.992170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8020.664684] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8024.482089] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8026.249695] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8026.331460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8027.624331] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8027.693526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8030.178118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8030.178285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.180525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.181059] LustreError: Skipped 1 previous similar message [ 8030.185273] Lustre: Skipped 1 previous similar message [ 8034.450211] Lustre: server umount lustre-OST0000 complete [ 8039.392067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8039.394806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.398853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.401920] Lustre: Skipped 1 previous similar message [ 8040.416425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8040.417080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.077690] Lustre: server umount lustre-MDT0000 complete [ 8043.737281] LustreError: 523825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100874 with bad export cookie 16417806053485545974 [ 8043.743648] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8043.745332] LustreError: 523825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8043.877682] Lustre: server umount lustre-MDT0001 complete [ 8053.219113] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8057.268456] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8069.773386] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7878 sec ========== 03:34:59 (1784100899) [ 8070.335707] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:35:00 (1784100900) === [ 8071.477129] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:35:01 (1784100901) === [ 8087.497934] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8088.465555] Key type lgssc unregistered [ 8088.579264] LNet: 530455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8088.581558] LNetError: 530455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8088.589265] LNet: Removed LNI 192.168.204.145@tcp [ 8088.882331] Key type .llcrypt unregistered [ 8088.883605] Key type ._llcrypt unregistered