[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 448120351 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002405] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008017] pid_max: default: 32768 minimum: 301 [ 0.009174] LSM: Security Framework initializing [ 0.011033] Yama: becoming mindful. [ 0.012048] SELinux: Initializing. [ 0.013095] *** VALIDATE selinux *** [ 0.021653] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026170] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031066] *** VALIDATE tmpfs *** [ 0.032524] *** VALIDATE proc *** [ 0.033282] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042376] debug: unmapping init [mem 0xffffffff9e259000-0xffffffff9e260fff] [ 0.044161] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045708] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.048834] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050009] ... fixed-purpose events: 3 [ 0.050964] ... event mask: 000000070000000f [ 0.051382] rcu: Hierarchical SRCU implementation. [ 0.053603] smp: Bringing up secondary CPUs ... [ 0.054785] x86: Booting SMP configuration: [ 0.055046] .... node #0, CPUs: #1 #2 #3 [ 0.059289] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.225031] node 0 deferred pages initialised in 161ms [ 0.228169] devtmpfs: initialized [ 0.229217] x86/mm: Memory block size: 128MB [ 0.231642] gcov: version magic: 0x41383552 [ 0.233359] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.234109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.235359] pinctrl core: initialized pinctrl subsystem [ 0.236204] [ 0.236768] ************************************************************* [ 0.237013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238015] ** ** [ 0.239020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.240017] ** ** [ 0.241018] ** This means that this kernel is built to expose internal ** [ 0.242014] ** IOMMU data structures, which may compromise security on ** [ 0.243016] ** your system. ** [ 0.244020] ** ** [ 0.245016] ** If you see this message and you are not debugging the ** [ 0.246014] ** kernel, report this immediately to your vendor! ** [ 0.247018] ** ** [ 0.248018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249014] ************************************************************* [ 0.250728] NET: Registered protocol family 16 [ 0.251474] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.252074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.253081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.255051] cpuidle: using governor menu [ 0.256920] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.258514] PCI: Using configuration type 1 for base access [ 0.260104] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.269125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.272024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.275169] cryptd: max_cpu_qlen set to 1000 [ 0.278926] ACPI: Added _OSI(Module Device) [ 0.280015] ACPI: Added _OSI(Processor Device) [ 0.281011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.283017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.287801] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.293467] ACPI: Interpreter enabled [ 0.295049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.296010] ACPI: Using IOAPIC for interrupt routing [ 0.297115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.300422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.309594] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.312052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.315022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.318094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.323401] acpiphp: Slot [2] registered [ 0.324096] acpiphp: Slot [5] registered [ 0.325109] acpiphp: Slot [6] registered [ 0.327128] acpiphp: Slot [7] registered [ 0.328072] acpiphp: Slot [8] registered [ 0.329125] acpiphp: Slot [9] registered [ 0.331129] acpiphp: Slot [10] registered [ 0.333161] acpiphp: Slot [3] registered [ 0.334117] acpiphp: Slot [4] registered [ 0.336090] acpiphp: Slot [11] registered [ 0.337170] acpiphp: Slot [12] registered [ 0.339115] acpiphp: Slot [13] registered [ 0.340141] acpiphp: Slot [14] registered [ 0.342108] acpiphp: Slot [15] registered [ 0.343118] acpiphp: Slot [16] registered [ 0.345164] acpiphp: Slot [17] registered [ 0.346089] acpiphp: Slot [18] registered [ 0.348089] acpiphp: Slot [19] registered [ 0.349122] acpiphp: Slot [20] registered [ 0.350086] acpiphp: Slot [21] registered [ 0.351084] acpiphp: Slot [22] registered [ 0.352085] acpiphp: Slot [23] registered [ 0.354080] acpiphp: Slot [24] registered [ 0.355081] acpiphp: Slot [25] registered [ 0.356115] acpiphp: Slot [26] registered [ 0.357104] acpiphp: Slot [27] registered [ 0.359081] acpiphp: Slot [28] registered [ 0.360127] acpiphp: Slot [29] registered [ 0.362083] acpiphp: Slot [30] registered [ 0.363085] acpiphp: Slot [31] registered [ 0.364097] PCI host bridge to bus 0000:00 [ 0.365019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.367028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.369022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.372027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.374022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.376029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.378203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.381000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.383180] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.392019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.397564] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.401019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.403019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.406427] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.409713] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.412046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.414859] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.418823] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.428020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.432013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440041] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.454019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.463021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.492000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.498017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.504016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.518023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.530153] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.537018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.545018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.576442] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.585015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.592018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.610020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.622831] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.631017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.638017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.656016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667885] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.673012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.678016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.696025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.707752] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.710374] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.713950] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.716469] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.719292] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.724148] iommu: Default domain type: Passthrough [ 0.725000] SCSI subsystem initialized [ 0.725166] ACPI: bus type USB registered [ 0.727113] usbcore: registered new interface driver usbfs [ 0.729104] usbcore: registered new interface driver hub [ 0.731180] usbcore: registered new device driver usb [ 0.733386] pps_core: LinuxPPS API ver. 1 registered [ 0.734007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.737076] PTP clock support registered [ 0.739190] EDAC MC: Ver: 3.0.0 [ 0.740449] PCI: Using ACPI for IRQ routing [ 0.742909] NetLabel: Initializing [ 0.744010] NetLabel: domain hash size = 128 [ 0.745008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.748092] NetLabel: unlabeled traffic allowed by default [ 0.750116] vgaarb: loaded [ 0.751351] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.754019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.763005] clocksource: Switched to clocksource kvm-clock [ 0.864445] VFS: Disk quotas dquot_6.6.0 [ 0.866728] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.869804] *** VALIDATE ramfs *** [ 0.871434] *** VALIDATE hugetlbfs *** [ 0.873410] pnp: PnP ACPI init [ 0.876323] pnp: PnP ACPI: found 6 devices [ 0.892140] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.894790] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.896617] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.898498] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.900447] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.902507] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.905151] NET: Registered protocol family 2 [ 0.907383] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.911707] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.915516] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.920756] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.923938] TCP: Hash tables configured (established 65536 bind 65536) [ 0.926375] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.929670] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.932097] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.934865] NET: Registered protocol family 1 [ 0.937461] RPC: Registered named UNIX socket transport module. [ 0.939420] RPC: Registered udp transport module. [ 0.941461] RPC: Registered tcp transport module. [ 0.943627] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.946356] NET: Registered protocol family 44 [ 0.947856] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.949778] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.952348] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.955352] PCI: CLS 0 bytes, default 64 [ 0.957489] Unpacking initramfs... [ 2.290988] debug: unmapping init [mem 0xffff97ad7cc54000-0xffff97ad7ffbffff] [ 2.295040] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.297160] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.300492] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.775112] Initialise system trusted keyrings [ 2.776712] Key type blacklist registered [ 2.778457] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.787471] zbud: loaded [ 2.790659] *** VALIDATE nfs *** [ 2.792310] *** VALIDATE nfs4 *** [ 2.794206] pstore: using deflate compression [ 2.797226] Platform Keyring initialized [ 2.908437] NET: Registered protocol family 38 [ 2.911058] Key type asymmetric registered [ 2.912875] Asymmetric key parser 'x509' registered [ 2.915533] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.919598] io scheduler mq-deadline registered [ 2.921311] io scheduler kyber registered [ 2.923280] io scheduler bfq registered [ 2.925879] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.929638] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.933293] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.936664] ACPI: Power Button [PWRF] [ 2.943074] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.951667] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.963204] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.969397] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.986303] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.012306] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.040045] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.044778] Non-volatile memory driver v1.3 [ 3.046322] Linux agpgart interface v0.103 [ 3.078240] virtio_blk virtio1: [vda] 136584 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.081659] vda: detected capacity change from 0 to 69931008 [ 3.093730] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.096176] vdb: detected capacity change from 0 to 1073741824 [ 3.110338] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.113249] vdc: detected capacity change from 0 to 2621440000 [ 3.131071] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.134919] vdd: detected capacity change from 0 to 2621440000 [ 3.148924] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.151141] vde: detected capacity change from 0 to 4294967296 [ 3.163445] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.165735] vdf: detected capacity change from 0 to 4294967296 [ 3.170988] libphy: Fixed MDIO Bus: probed [ 3.175650] usbcore: registered new interface driver usbserial_generic [ 3.177541] usbserial: USB Serial support registered for generic [ 3.179361] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.183052] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.184630] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.186724] mousedev: PS/2 mouse device common for all mice [ 3.189433] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.191824] rtc_cmos 00:05: RTC can wake from S4 [ 3.195936] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.198234] rtc_cmos 00:05: registered as rtc0 [ 3.203434] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.204074] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.211822] intel_pstate: CPU model not supported [ 3.215604] hid: raw HID events driver (C) Jiri Kosina [ 3.218205] usbcore: registered new interface driver usbhid [ 3.219736] usbhid: USB HID core driver [ 3.221076] drop_monitor: Initializing network drop monitor service [ 3.223155] Initializing XFRM netlink socket [ 3.224746] NET: Registered protocol family 10 [ 3.227565] Segment Routing with IPv6 [ 3.228559] NET: Registered protocol family 17 [ 3.229796] mpls_gso: MPLS GSO support [ 3.235885] RAS: Correctable Errors collector initialized. [ 3.238067] AVX version of gcm_enc/dec engaged. [ 3.240233] AES CTR mode by8 optimization enabled [ 3.309273] sched_clock: Marking stable (3309250864, 0)->(4188755453, -879504589) [ 3.312913] registered taskstats version 1 [ 3.314925] Loading compiled-in X.509 certificates [ 3.316756] zswap: loaded using pool lzo/zbud [ 3.338078] Key type big_key registered [ 3.351337] Key type encrypted registered [ 3.352882] ima: No TPM chip found, activating TPM-bypass! [ 3.354954] ima: Allocated hash algorithm: sha1 [ 3.356643] ima: No architecture policies found [ 3.358710] evm: Initialising EVM extended attributes: [ 3.360711] evm: security.selinux [ 3.361719] evm: security.ima [ 3.362439] evm: security.capability [ 3.363452] evm: HMAC attrs: 0x1 [ 3.365454] rtc_cmos 00:05: setting system clock to 2026-06-12 03:48:06 UTC (1781236086) [ 3.371740] debug: unmapping init [mem 0xffffffff9f203000-0xffffffff9f3fffff] [ 3.374900] debug: unmapping init [mem 0xffffffff9df82000-0xffffffff9e258fff] [ 3.384153] Write protecting the kernel read-only data: 28672k [ 3.388417] debug: unmapping init [mem 0xffffffff9c603000-0xffffffff9c7fffff] [ 3.391753] debug: unmapping init [mem 0xffffffff9cf14000-0xffffffff9cffffff] [ 3.421722] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.429043] systemd[1]: Detected virtualization kvm. [ 3.430955] systemd[1]: Detected architecture x86-64. [ 3.432483] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.460113] systemd[1]: No hostname configured. [ 3.461811] systemd[1]: Set hostname to . [ 3.463323] random: systemd: uninitialized urandom read (16 bytes read) [ 3.465738] systemd[1]: Initializing machine ID from random generator. [ 3.516495] random: ln: uninitialized urandom read (6 bytes read) [ 3.599546] random: systemd: uninitialized urandom read (16 bytes read) [ 3.604281] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.612648] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.616573] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.245440] device-mapper: uevent: version 1.0.3 [ 4.247392] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [ 4.894850] random: fast init done ] Reached target Basic System. [ 4.909600] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.984233] scsi host0: ata_piix [ 5.005044] scsi host1: ata_piix [ 5.006796] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.009221] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.508449] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.868765] random: crng init done [ 9.870330] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.317274] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.386902] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.635385] SELinux: Disabled at runtime. [ 11.700365] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.709897] systemd[1]: Detected virtualization kvm. [ 11.712330] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.195496] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.198893] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.203166] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.207908] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.211119] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.218280] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.225485] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slic[ 12.256790] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.607657] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.943563] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.000562] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.089141] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.109319] EDAC sbridge: Ver: 1.1.2 [ 14.493806] Key type dns_resolver registered [ 14.793339] NFS: Registering the id_resolver key type [ 14.795651] Key type id_resolver registered [ 14.797424] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg607-server login: [ 41.096529] libcfs: loading out-of-tree module taints kernel. [ 41.119635] Key type ._llcrypt registered [ 41.121503] Key type .llcrypt registered [ 41.177496] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 50.230241] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 50.978689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.987830] alg: No test for adler32 (adler32-zlib) [ 52.050702] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 52.428128] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 54.063170] Key type lgssc registered [ 54.809277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 62.420375] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.403644] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 85.950375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 85.966239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.092224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 87.119693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 87.172246] Lustre: lustre-MDT0000: new disk, initializing [ 87.233697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.245581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 88.962337] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.842547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.886843] Lustre: 6477:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 94.903800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 94.906437] Lustre: Skipped 1 previous similar message [ 94.947393] Lustre: lustre-MDT0001: new disk, initializing [ 94.975641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 94.993044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 95.001528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 96.584730] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.175731] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 103.034854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 103.152748] Lustre: lustre-OST0000: new disk, initializing [ 103.155704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 103.159020] Lustre: 8382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 103.183880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 105.710144] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.096440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 111.101289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 111.128641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 112.376650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 112.449088] Lustre: lustre-OST0001: new disk, initializing [ 112.451546] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 112.454241] Lustre: 9435:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 112.481670] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 114.912866] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 119.798937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 119.804205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 119.828207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 121.570129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 125.746237] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 128.326057] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing check_logdir /tmp/testlogs/ [ 130.194739] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing yml_node [ 132.103826] Lustre: DEBUG MARKER: Client: 2.17.53.79 [ 133.186102] Lustre: DEBUG MARKER: MDS: 2.17.53.79 [ 134.253754] Lustre: DEBUG MARKER: OSS: 2.17.53.79 [ 134.918166] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 23:50:18 EDT 2026 [ 142.176534] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 142.888207] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 155.616211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.616838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 155.621604] Lustre: Skipped 1 previous similar message [ 155.626645] Lustre: Skipped 3 previous similar messages [ 160.737718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 160.743667] Lustre: Skipped 3 previous similar messages [ 161.198426] Lustre: server umount lustre-MDT0000 complete [ 162.876831] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236246 with bad export cookie 12077899153006874655 [ 162.880154] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 162.882550] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 165.855812] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.856512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 165.863490] Lustre: Skipped 3 previous similar messages [ 167.905409] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 167.909489] Lustre: Skipped 2 previous similar messages [ 169.089391] Lustre: server umount lustre-MDT0001 complete [ 185.311405] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 185.371554] Lustre: server umount lustre-OST0000 complete [ 201.183142] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 201.245564] Lustre: server umount lustre-OST0001 complete [ 204.043775] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 207.687792] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 223.693196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.824036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 223.845543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 223.886275] Lustre: lustre-MDT0000: new disk, initializing [ 223.934410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.948927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 225.793420] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 230.913222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.984506] Lustre: 14940:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.153487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.855725] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 236.551345] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 238.114989] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 241.061088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.183631] Lustre: lustre-OST0000: new disk, initializing [ 241.187100] Lustre: Skipped 1 previous similar message [ 241.189913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.193529] Lustre: Skipped 2 previous similar messages [ 241.196796] Lustre: 16684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.238255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.242801] Lustre: Skipped 1 previous similar message [ 242.964317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.969895] Lustre: Skipped 1 previous similar message [ 242.972445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.004739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.766182] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 247.573571] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 249.402282] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 249.496520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 251.233683] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 251.327777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 253.412397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 253.413061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.414198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 253.417506] LustreError: Skipped 1 previous similar message [ 253.427669] Lustre: Skipped 1 previous similar message [ 258.475241] Lustre: server umount lustre-OST0000 complete [ 262.112665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 262.117431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.123506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.126505] Lustre: Skipped 1 previous similar message [ 274.399129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 274.485911] Lustre: server umount lustre-MDT0000 complete [ 276.220133] LustreError: 14933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236359 with bad export cookie 12077899153006876181 [ 276.223351] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 276.226207] LustreError: 14933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 276.392994] Lustre: server umount lustre-MDT0001 complete [ 278.043477] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 23:52:41 (1781236361) [ 281.122588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 281.378085] LustreError: 18798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.393353] LustreError: 18798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 281.424641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 283.222118] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 286.688790] LustreError: 18799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.958643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 288.772181] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 290.998367] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 292.444769] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 294.991138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 297.316969] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 299.566167] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 312.800449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 312.805694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.811979] Lustre: Skipped 1 previous similar message [ 312.815152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 312.818133] Lustre: Skipped 6 previous similar messages [ 316.898439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 318.935930] Lustre: server umount lustre-OST0000 complete [ 322.017047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.023026] Lustre: Skipped 2 previous similar messages [ 326.620555] Lustre: server umount lustre-MDT0000 complete [ 327.136065] LustreError: 18798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.142500] LustreError: 18798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 328.084175] LustreError: 18780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236411 with bad export cookie 12077899153006877070 [ 328.088027] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.089442] LustreError: 18780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 328.222824] Lustre: server umount lustre-MDT0001 complete [ 332.770806] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 334.081745] Key type lgssc unregistered [ 334.253120] LNet: 21992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.258128] LNetError: 21992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 334.267547] LNet: Removed LNI 192.168.206.107@tcp [ 334.768141] Key type .llcrypt unregistered [ 334.770632] Key type ._llcrypt unregistered [ 346.119949] Key type ._llcrypt registered [ 346.121868] Key type .llcrypt registered [ 346.183368] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 346.566073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 346.600482] alg: No test for adler32 (adler32-zlib) [ 347.476425] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 347.574328] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 349.159271] Key type lgssc registered [ 349.629264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 353.222805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 353.245830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.512394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.335673] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 360.412455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.587966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 362.396850] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 365.256983] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 367.092283] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 370.670426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 370.838495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.730878] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 376.103292] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 379.369068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 384.483222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 384.484323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.485334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 384.494063] Lustre: Skipped 1 previous similar message [ 388.443525] Lustre: server umount lustre-OST0000 complete [ 391.650062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 391.653519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.659736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 391.662947] Lustre: Skipped 1 previous similar message [ 394.720953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.721401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.447599] Lustre: server umount lustre-MDT0000 complete [ 397.944409] LustreError: 23348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236481 with bad export cookie 5345449713326817625 [ 397.947088] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 397.949665] LustreError: 23348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 398.077848] Lustre: server umount lustre-MDT0001 complete [ 402.428884] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 403.740176] Key type lgssc unregistered [ 403.893564] LNet: 26982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.897535] LNetError: 26982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 403.908411] LNet: Removed LNI 192.168.206.107@tcp [ 404.271951] Key type .llcrypt unregistered [ 404.276705] Key type ._llcrypt unregistered [ 414.990631] Key type ._llcrypt registered [ 414.992090] Key type .llcrypt registered [ 415.040394] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 415.567416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 415.578063] alg: No test for adler32 (adler32-zlib) [ 416.485204] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 416.587735] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 418.191144] Key type lgssc registered [ 418.704383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 422.256963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 422.267291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.491804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.208693] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 428.967454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.144751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.783753] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 433.153789] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 434.758580] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 440.181700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.353837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.854690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 445.150184] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 448.551584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 458.720318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 458.724267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.729787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.746459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.746468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 459.746533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.479452] Lustre: server umount lustre-OST0000 complete [ 469.985768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 469.986084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.988477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 469.988488] Lustre: Skipped 1 previous similar message [ 469.998502] Lustre: Skipped 1 previous similar message [ 472.449894] Lustre: server umount lustre-MDT0000 complete [ 474.055171] LustreError: 28336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236557 with bad export cookie 11591241401684851899 [ 474.057436] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.063213] LustreError: 28336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 474.248815] Lustre: server umount lustre-MDT0001 complete [ 478.230101] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 479.408806] Key type lgssc unregistered [ 479.544508] LNet: 31891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 479.547957] LNetError: 31891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 479.557312] LNet: Removed LNI 192.168.206.107@tcp [ 479.897213] Key type .llcrypt unregistered [ 479.898536] Key type ._llcrypt unregistered [ 490.052142] Key type ._llcrypt registered [ 490.053750] Key type .llcrypt registered [ 490.103255] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 490.494917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 490.509458] alg: No test for adler32 (adler32-zlib) [ 491.412621] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 491.533161] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 493.143189] Key type lgssc registered [ 493.628356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 497.097848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 497.106422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.342880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.106761] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 503.914144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.095186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 505.991830] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 508.412679] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 509.932873] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 512.868909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 513.032449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.630630] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 517.980857] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 523.305302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 528.357704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.357763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.360152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.363856] Lustre: Skipped 1 previous similar message [ 529.751675] Lustre: server umount lustre-OST0000 complete [ 533.401497] Lustre: server umount lustre-MDT0000 complete [ 533.473210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.474404] LustreError: 33264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.478736] Lustre: Skipped 1 previous similar message [ 533.484454] LustreError: 33264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 534.937386] LustreError: 33245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236618 with bad export cookie 11561252002648379971 [ 534.940995] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 534.943407] LustreError: 33245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 535.077729] Lustre: server umount lustre-MDT0001 complete [ 539.408675] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 540.543683] Key type lgssc unregistered [ 540.684522] LNet: 36434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.688587] LNetError: 36434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.696367] LNet: Removed LNI 192.168.206.107@tcp [ 541.030087] Key type .llcrypt unregistered [ 541.031420] Key type ._llcrypt unregistered [ 551.265325] Key type ._llcrypt registered [ 551.266872] Key type .llcrypt registered [ 551.311477] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 551.681162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 551.748968] alg: No test for adler32 (adler32-zlib) [ 552.630157] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 552.732700] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 554.319166] Key type lgssc registered [ 554.691942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 557.606349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 557.613505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.780857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.088246] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 563.157881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.290971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 564.656311] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 566.611265] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 567.953370] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 570.677035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.834855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.749056] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 576.286966] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 584.172075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 592.176822] Lustre: lustre-OST0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 593.376216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 593.379724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.386307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.402087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.403353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.405348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.281946] Lustre: lustre-OST0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 597.286294] Lustre: Skipped 1 previous similar message [ 602.401332] Lustre: lustre-OST0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 602.405223] Lustre: Skipped 2 previous similar messages [ 604.127136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 604.182280] Lustre: server umount lustre-OST0000 complete [ 617.913149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.922248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 617.930558] Lustre: Skipped 1 previous similar message [ 632.287167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 632.349686] Lustre: server umount lustre-MDT0000 complete [ 633.942621] LustreError: 37786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236717 with bad export cookie 7322881817682288296 [ 633.947306] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.947963] LustreError: 37786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 640.154122] Lustre: server umount lustre-MDT0001 complete [ 644.349356] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 645.472794] Key type lgssc unregistered [ 645.618407] LNet: 41176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.621459] LNetError: 41176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.632467] LNet: Removed LNI 192.168.206.107@tcp [ 645.993590] Key type .llcrypt unregistered [ 645.995177] Key type ._llcrypt unregistered [ 656.040194] Key type ._llcrypt registered [ 656.041939] Key type .llcrypt registered [ 656.085633] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 656.546646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 656.561536] alg: No test for adler32 (adler32-zlib) [ 657.444331] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 657.554857] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 659.151171] Key type lgssc registered [ 659.665154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 662.914546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 662.924066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.151929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.773809] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 669.455986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.626547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 671.296863] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 673.529496] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 674.986960] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 677.602625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.746953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.054018] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 682.282449] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 689.000351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 698.159394] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.7@tcp (stopping) [ 699.360621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.370219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.372807] Lustre: Skipped 1 previous similar message [ 701.727444] Lustre: server umount lustre-MDT0000 complete [ 703.227263] LustreError: 42531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236786 with bad export cookie 6938756342568221861 [ 703.229764] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.234698] LustreError: 42531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 703.265281] LustreError: 44430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.273121] LustreError: 44430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 703.277495] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.7@tcp (stopping) [ 704.481239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.481662] LustreError: 44430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.486810] Lustre: Skipped 2 previous similar messages [ 708.385481] LustreError: 42548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.386062] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.7@tcp (stopping) [ 708.398415] Lustre: Skipped 1 previous similar message [ 709.432550] Lustre: server umount lustre-MDT0001 complete [ 727.723321] Lustre: server umount lustre-OST0000 complete [ 735.168078] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 736.461421] Key type lgssc unregistered [ 736.629190] LNet: 46072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.633179] LNetError: 46072:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.644449] LNet: Removed LNI 192.168.206.107@tcp [ 737.027845] Key type .llcrypt unregistered [ 737.029489] Key type ._llcrypt unregistered [ 748.711628] Key type ._llcrypt registered [ 748.714772] Key type .llcrypt registered [ 748.769897] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 749.159573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 749.169392] alg: No test for adler32 (adler32-zlib) [ 750.070676] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 750.199693] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 751.799124] Key type lgssc registered [ 752.332701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 755.912709] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 755.924956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 771.359412] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 776.479411] LustreError: 47419:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 803.103183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.237315] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 807.242831] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 972.233634] Lustre: server umount lustre-OST0000 complete [ 979.112952] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 980.488963] Key type lgssc unregistered [ 980.650476] LNet: 48976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.658074] LNetError: 48976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.670150] LNet: Removed LNI 192.168.206.107@tcp [ 981.036262] Key type .llcrypt unregistered [ 981.038886] Key type ._llcrypt unregistered [ 991.894142] Key type ._llcrypt registered [ 991.895873] Key type .llcrypt registered [ 991.948095] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 992.421117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.433549] alg: No test for adler32 (adler32-zlib) [ 993.329365] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 993.435026] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 995.023190] Key type lgssc registered [ 995.468760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 998.840637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 998.850267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.099075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.730394] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1005.740783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.956364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1007.679367] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1010.194837] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.755930] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1014.739547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1014.914450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.367748] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1019.812672] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1021.412306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.412555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1021.418762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.875203] Lustre: server umount lustre-OST0000 complete [ 1024.476253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.477173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1024.482573] Lustre: Skipped 1 previous similar message [ 1029.600704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.603113] Lustre: Skipped 2 previous similar messages [ 1030.623711] Lustre: server umount lustre-MDT0000 complete [ 1032.065509] LustreError: 50328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237115 with bad export cookie 12004947858734755344 [ 1032.069189] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1032.070733] LustreError: 50328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1032.193198] Lustre: server umount lustre-MDT0001 complete [ 1036.138141] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1037.333686] Key type lgssc unregistered [ 1037.489631] LNet: 53520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.492438] LNetError: 53520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.501587] LNet: Removed LNI 192.168.206.107@tcp [ 1037.879044] Key type .llcrypt unregistered [ 1037.880973] Key type ._llcrypt unregistered [ 1048.136263] Key type ._llcrypt registered [ 1048.137848] Key type .llcrypt registered [ 1048.196800] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1048.729687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.759823] alg: No test for adler32 (adler32-zlib) [ 1049.646493] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1049.757334] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1051.343192] Key type lgssc registered [ 1051.865077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1055.458524] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1055.469812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1070.943917] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1076.063408] LustreError: 54866:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 1102.687182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.961885] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1107.177994] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1110.325405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.567918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.600149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1112.472682] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1115.617430] LustreError: 54891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.625500] LustreError: 54891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1116.281202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.470403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1118.118386] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1120.402636] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1121.922396] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1125.856162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1125.860085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.868981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1126.881515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1126.884822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.890948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.906055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.908927] Lustre: Skipped 1 previous similar message [ 1133.025946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1133.029273] Lustre: Skipped 1 previous similar message [ 1137.119149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1137.179267] Lustre: server umount lustre-OST0000 complete [ 1152.482331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1152.487882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.498165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.308628] Lustre: server umount lustre-MDT0000 complete [ 1158.624586] LustreError: 55792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.631472] LustreError: 55792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1159.861695] LustreError: 57516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237242 with bad export cookie 13446118639861462483 [ 1159.864343] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.867159] LustreError: 57516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1166.076026] Lustre: server umount lustre-MDT0001 complete [ 1170.329961] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1171.621230] Key type lgssc unregistered [ 1171.760411] LNet: 58250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.763837] LNetError: 58250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.774466] LNet: Removed LNI 192.168.206.107@tcp [ 1172.172280] Key type .llcrypt unregistered [ 1172.174049] Key type ._llcrypt unregistered [ 1182.495449] Key type ._llcrypt registered [ 1182.497287] Key type .llcrypt registered [ 1182.547732] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1182.837693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1182.869867] alg: No test for adler32 (adler32-zlib) [ 1183.743656] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1183.836873] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1185.431133] Key type lgssc registered [ 1185.873773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1188.917247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1188.932272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.150102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.823234] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1195.459112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.654150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1197.326607] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1199.632852] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1201.303349] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1204.352345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.518217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.901945] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1209.152927] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1215.786035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1226.209416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1226.211651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.215047] LustreError: Skipped 1 previous similar message [ 1226.222603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1226.228471] Lustre: Skipped 1 previous similar message [ 1229.147389] Lustre: server umount lustre-OST0000 complete [ 1231.328946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.330941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.338925] Lustre: Skipped 2 previous similar messages [ 1231.345706] Lustre: Skipped 1 previous similar message [ 1236.449828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.088788] Lustre: server umount lustre-MDT0000 complete [ 1238.558381] LustreError: 59614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237321 with bad export cookie 18375562055078631612 [ 1238.560524] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1238.564262] LustreError: 59614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1238.694309] Lustre: server umount lustre-MDT0001 complete [ 1242.750562] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1244.000333] Key type lgssc unregistered [ 1244.147782] LNet: 62826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.151709] LNetError: 62826:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.162496] LNet: Removed LNI 192.168.206.107@tcp [ 1244.524417] Key type .llcrypt unregistered [ 1244.525932] Key type ._llcrypt unregistered [ 1255.753198] Key type ._llcrypt registered [ 1255.754684] Key type .llcrypt registered [ 1255.795875] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1256.216019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1256.231284] alg: No test for adler32 (adler32-zlib) [ 1257.099823] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1257.200292] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1258.791214] Key type lgssc registered [ 1259.215665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1262.664169] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 00:09:06 (1781237346) [ 1265.090072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1265.097128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.262320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.638700] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1270.798395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.961444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1272.503457] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1274.667920] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1276.147934] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1279.181495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1279.347956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.646240] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1283.987950] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1291.622458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1298.340689] Lustre: Failing over lustre-MDT0000 [ 1298.506633] Lustre: server umount lustre-MDT0000 complete [ 1300.790299] LustreError: 64486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.803704] LustreError: 64486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1301.985280] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.989445] LustreError: 66371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.990834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1301.995157] Lustre: Skipped 2 previous similar messages [ 1302.000152] LustreError: 66371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1302.987864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.059400] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.141725] Lustre: *** cfs_fail_loc=135, val=0*** [ 1303.188843] LustreError: 67257:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1303.194377] LustreError: 67257:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.107@tcp: cfg command failed: rc = -2 [ 1303.198439] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1303.198439] [ 1303.202480] LustreError: MGC192.168.206.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1303.211612] LustreError: 67245:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1303.214585] LustreError: 67245:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1303.218229] LustreError: 67245:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1303.220426] LustreError: 67245:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1303.234497] Lustre: server umount lustre-MDT0000 complete [ 1303.236630] LustreError: 67245:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1305.892139] LustreError: 64514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.899507] LustreError: 64514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1306.230457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.293537] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.427599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.457247] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1308.147137] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1311.009665] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1311.718312] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1311.731260] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1311.750649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1316.833272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1316.834099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.838874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1316.844627] Lustre: Skipped 1 previous similar message [ 1318.931288] Lustre: server umount lustre-OST0000 complete [ 1321.952870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.953477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1321.957802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.957810] Lustre: Skipped 1 previous similar message [ 1321.961151] Lustre: Skipped 1 previous similar message [ 1326.567828] Lustre: server umount lustre-MDT0000 complete [ 1327.077721] LustreError: 64514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.088256] LustreError: 64514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1328.382401] LustreError: 64469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237411 with bad export cookie 2838944902999209685 [ 1328.386661] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.389803] LustreError: 64469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1328.554599] Lustre: server umount lustre-MDT0001 complete [ 1333.181383] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1334.353858] Key type lgssc unregistered [ 1334.496484] LNet: 68998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.502684] LNetError: 68998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.511334] LNet: Removed LNI 192.168.206.107@tcp [ 1334.850836] Key type .llcrypt unregistered [ 1334.852437] Key type ._llcrypt unregistered [ 1344.985262] Key type ._llcrypt registered [ 1344.986993] Key type .llcrypt registered [ 1345.033196] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1345.443442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1345.482991] alg: No test for adler32 (adler32-zlib) [ 1346.361047] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1346.463405] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1348.055427] Key type lgssc registered [ 1348.519810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.652965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.662044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.856277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.474592] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1358.338914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.531713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1360.380904] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1362.993905] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1364.679653] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1367.996967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1368.172280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.689476] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1373.200537] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1377.384388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1380.907576] Lustre: Failing over lustre-MDT0000 [ 1381.037368] Lustre: server umount lustre-MDT0000 complete [ 1382.882744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.885868] LustreError: 70366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.892715] Lustre: Skipped 2 previous similar messages [ 1382.903705] LustreError: 70366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1383.729733] LustreError: 71962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.736528] LustreError: 71962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1385.682051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.786947] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.853047] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1385.918572] LustreError: 73116:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1385.923356] LustreError: 73116:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.107@tcp: cfg command failed: rc = -9 [ 1385.928412] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1385.928412] [ 1385.933949] LustreError: MGC192.168.206.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1385.942301] LustreError: 73104:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1385.946457] LustreError: 73104:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 1385.949889] LustreError: 73104:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1385.955089] LustreError: 73104:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1385.971210] Lustre: server umount lustre-MDT0000 complete [ 1385.973407] LustreError: 73104:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1388.001934] LustreError: 70365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.014166] LustreError: 70365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1389.519267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.622182] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1389.791524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1389.824844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1391.732955] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1393.955111] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1395.175715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1395.189510] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1395.216456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1400.289947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.293134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1400.295701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1400.296329] Lustre: Skipped 1 previous similar message [ 1400.306130] Lustre: Skipped 1 previous similar message [ 1403.033614] Lustre: server umount lustre-OST0000 complete [ 1405.409451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1405.410396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.411755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1405.414622] LustreError: Skipped 1 previous similar message [ 1405.428416] Lustre: Skipped 1 previous similar message [ 1410.528437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.536386] Lustre: Skipped 2 previous similar messages [ 1415.654022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1415.659658] Lustre: Skipped 1 previous similar message [ 1419.231124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1419.297543] Lustre: server umount lustre-MDT0000 complete [ 1420.769760] LustreError: 70367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.780206] LustreError: 70367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1421.105056] LustreError: 70352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237504 with bad export cookie 7539562665934291602 [ 1421.107478] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1421.110793] LustreError: 70352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1421.281620] Lustre: server umount lustre-MDT0001 complete [ 1426.026381] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1427.308527] Key type lgssc unregistered [ 1427.444406] LNet: 74860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.448107] LNetError: 74860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.463134] LNet: Removed LNI 192.168.206.107@tcp [ 1427.804804] Key type .llcrypt unregistered [ 1427.806493] Key type ._llcrypt unregistered [ 1432.422284] Key type ._llcrypt registered [ 1432.423835] Key type .llcrypt registered [ 1432.470101] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1432.586316] Key type .llcrypt unregistered [ 1432.588705] Key type ._llcrypt unregistered [ 1442.837369] Key type ._llcrypt registered [ 1442.841022] Key type .llcrypt registered [ 1442.892227] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1443.307533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1443.329557] alg: No test for adler32 (adler32-zlib) [ 1444.225264] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1444.326168] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1445.927203] Key type lgssc registered [ 1446.396152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1450.285109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1450.298282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.545114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.352497] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1457.592061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.774667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1459.603700] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1462.275910] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1464.257899] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1467.440622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.618616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.277224] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1472.705603] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1476.841169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1492.448307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1492.452240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.456239] LustreError: Skipped 1 previous similar message [ 1492.463686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.468533] Lustre: Skipped 1 previous similar message [ 1493.468829] Lustre: server umount lustre-OST0000 complete [ 1497.571773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.579238] Lustre: Skipped 1 previous similar message [ 1497.582174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1502.688476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1502.691420] Lustre: Skipped 2 previous similar messages [ 1507.808933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.811494] Lustre: Skipped 1 previous similar message [ 1509.343115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1509.407673] Lustre: server umount lustre-MDT0000 complete [ 1510.785276] LustreError: 76684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237593 with bad export cookie 17243504895114097582 [ 1510.788141] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.789958] LustreError: 76684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1510.923072] Lustre: server umount lustre-MDT0001 complete [ 1515.164926] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1516.457790] Key type lgssc unregistered [ 1516.595462] LNet: 79898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.599668] LNetError: 79898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.608363] LNet: Removed LNI 192.168.206.107@tcp [ 1516.960437] Key type .llcrypt unregistered [ 1516.962330] Key type ._llcrypt unregistered [ 1527.029192] Key type ._llcrypt registered [ 1527.030779] Key type .llcrypt registered [ 1527.080703] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1527.536237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1527.551515] alg: No test for adler32 (adler32-zlib) [ 1528.451599] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1528.584724] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1530.191165] Key type lgssc registered [ 1530.633813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1534.108085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1534.119218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.321512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.904720] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1540.570228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.742458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1542.475737] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1544.627748] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1546.107461] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1548.734391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1548.904047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.936125] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1552.997929] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1561.194551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1570.784234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1570.791386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.798502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1571.808975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1571.813993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1571.817075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1573.037395] Lustre: server umount lustre-OST0000 complete [ 1576.929133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1576.933709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.935283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.944933] Lustre: Skipped 2 previous similar messages [ 1582.048358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1582.053887] Lustre: Skipped 1 previous similar message [ 1587.170993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.174648] Lustre: Skipped 1 previous similar message [ 1589.215143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1589.290588] Lustre: server umount lustre-MDT0000 complete [ 1590.729410] LustreError: 81263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237673 with bad export cookie 1548002946289552151 [ 1590.732447] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1590.734807] LustreError: 81263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1590.874755] Lustre: server umount lustre-MDT0001 complete [ 1595.292895] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1596.532060] Key type lgssc unregistered [ 1596.699634] LNet: 84478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.704914] LNetError: 84478:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.719665] LNet: Removed LNI 192.168.206.107@tcp [ 1597.077143] Key type .llcrypt unregistered [ 1597.078592] Key type ._llcrypt unregistered [ 1607.648516] Key type ._llcrypt registered [ 1607.650240] Key type .llcrypt registered [ 1607.696762] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1608.146667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.194285] alg: No test for adler32 (adler32-zlib) [ 1609.073074] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1609.182084] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1610.775365] Key type lgssc registered [ 1611.261117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1614.553732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1614.564888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.795452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.522516] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1621.155715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.327964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1622.924448] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1625.173724] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1626.634402] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1629.546290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1629.710843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.062159] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1634.361601] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1639.977824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1645.025183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.027557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.032124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1645.043564] Lustre: Skipped 1 previous similar message [ 1650.148919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1650.154237] Lustre: Skipped 2 previous similar messages [ 1650.338923] Lustre: server umount lustre-OST0000 complete [ 1652.203237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1652.211338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.219127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1655.264529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.265206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1655.274541] Lustre: Skipped 1 previous similar message [ 1658.163857] Lustre: server umount lustre-MDT0000 complete [ 1659.849490] LustreError: 88013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237742 with bad export cookie 2092518052270484250 [ 1659.852504] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1659.862123] LustreError: 88013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1659.996210] Lustre: server umount lustre-MDT0001 complete [ 1664.487795] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1665.733361] Key type lgssc unregistered [ 1665.883611] LNet: 89035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.887895] LNetError: 89035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.902520] LNet: Removed LNI 192.168.206.107@tcp [ 1666.282937] Key type .llcrypt unregistered [ 1666.284835] Key type ._llcrypt unregistered [ 1677.374160] Key type ._llcrypt registered [ 1677.376121] Key type .llcrypt registered [ 1677.442432] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1677.892945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.914416] alg: No test for adler32 (adler32-zlib) [ 1678.806463] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1678.918696] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1680.511180] Key type lgssc registered [ 1680.998053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1684.282414] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1684.291387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1699.743368] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1704.863321] LustreError: 90381:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 1731.487145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.863935] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1736.114049] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1738.278571] Lustre: server umount lustre-OST0000 complete [ 1741.510662] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 00:17:05 (1781237825) [ 1744.426034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.626144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.308928] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1749.984114] LustreError: 91904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1749.994585] LustreError: 91904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1750.284213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.481588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1752.442964] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1754.853360] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1756.348225] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1759.119199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.296477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.957184] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1764.791695] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1769.589264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1780.194190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.201527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.202178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.212795] Lustre: Skipped 1 previous similar message [ 1782.322412] Lustre: server umount lustre-OST0000 complete [ 1785.314810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.317320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.324199] Lustre: Skipped 2 previous similar messages [ 1785.331057] Lustre: Skipped 1 previous similar message [ 1790.432616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.435321] Lustre: Skipped 1 previous similar message [ 1790.584089] Lustre: server umount lustre-MDT0000 complete [ 1792.413692] LustreError: 94213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237875 with bad export cookie 6911792196636027486 [ 1792.416509] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.424841] LustreError: 94213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1792.590115] Lustre: server umount lustre-MDT0001 complete [ 1797.371801] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1798.729716] Key type lgssc unregistered [ 1798.884538] LNet: 95233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.889664] LNetError: 95233:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.898421] LNet: Removed LNI 192.168.206.107@tcp [ 1799.304201] Key type .llcrypt unregistered [ 1799.306562] Key type ._llcrypt unregistered [ 1818.354363] Key type ._llcrypt registered [ 1818.356125] Key type .llcrypt registered [ 1818.426356] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1818.834452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.872063] alg: No test for adler32 (adler32-zlib) [ 1819.792666] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1819.915356] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1821.535189] Key type lgssc registered [ 1822.019752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1825.477211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1825.489287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1826.723509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.467950] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1832.367815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.571765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.295175] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1836.484713] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1837.916095] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1840.756917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.920278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.607634] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1846.110407] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1851.179354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1856.480742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1856.487482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.497772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.134670] Lustre: server umount lustre-OST0000 complete [ 1861.603276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.607739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.614619] Lustre: Skipped 2 previous similar messages [ 1861.622320] Lustre: Skipped 1 previous similar message [ 1861.987209] Lustre: server umount lustre-MDT0000 complete [ 1863.604847] LustreError: 97357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237946 with bad export cookie 13460041079901820830 [ 1863.607174] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1863.613636] LustreError: 97357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1863.760689] Lustre: server umount lustre-MDT0001 complete [ 1868.294689] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 1869.643183] Key type lgssc unregistered [ 1869.789236] LNet: 100549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.792434] LNetError: 100549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.803630] LNet: Removed LNI 192.168.206.107@tcp [ 1870.186823] Key type .llcrypt unregistered [ 1870.189421] Key type ._llcrypt unregistered [ 1879.176638] Key type ._llcrypt registered [ 1879.178540] Key type .llcrypt registered [ 1879.239731] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1879.597460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1879.662466] alg: No test for adler32 (adler32-zlib) [ 1880.561444] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1880.681159] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 1882.287307] Key type lgssc registered [ 1882.762125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.060968] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.069617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1901.535336] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1906.656766] LustreError: 101753:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 1933.279195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.576380] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1939.729103] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1943.673924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.825570] LustreError: MGC192.168.206.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1943.832681] LustreError: 102663:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1943.836555] LustreError: 102663:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1943.840408] LustreError: 102663:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1943.884306] Lustre: server umount lustre-MDT0000 complete [ 1943.886052] LustreError: 102663:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1957.907691] Lustre: server umount lustre-OST0000 complete [ 1962.808523] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 1968.014549] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 1987.302205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.442505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1987.500578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1987.543428] Lustre: lustre-MDT0000: new disk, initializing [ 1987.590392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.600952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1989.444412] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 1995.215457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.288852] Lustre: 105417:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1995.321793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1995.325286] Lustre: Skipped 1 previous similar message [ 1995.372533] Lustre: lustre-MDT0001: new disk, initializing [ 1995.437166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1995.461133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1995.468800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1997.403858] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2001.284608] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2002.872954] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2006.255465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2006.421756] Lustre: lustre-OST0000: new disk, initializing [ 2006.425825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2006.429547] Lustre: 107157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2006.478746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.689092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2007.694310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2007.760186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2009.806858] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2014.293623] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2016.545922] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2016.669766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2018.689797] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2018.803341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2023.392848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.396832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.405507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2023.409204] Lustre: Skipped 1 previous similar message [ 2025.859755] Lustre: server umount lustre-OST0000 complete [ 2028.513908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.515062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2028.520248] Lustre: Skipped 2 previous similar messages [ 2033.632400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.635425] Lustre: Skipped 2 previous similar messages [ 2033.702859] Lustre: server umount lustre-MDT0000 complete [ 2035.472324] LustreError: 105411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238118 with bad export cookie 18230965897404543381 [ 2035.476432] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2035.482263] LustreError: 105411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2035.628415] Lustre: server umount lustre-MDT0001 complete [ 2039.261390] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 00:22:02 (1781238122) [ 2040.849454] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2047.779070] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 2052.118898] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2067.983667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.122622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2068.139218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2068.196420] Lustre: lustre-MDT0000: new disk, initializing [ 2068.253502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.265062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2070.249864] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2076.025652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.097714] Lustre: 111866:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2076.143130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2076.146576] Lustre: Skipped 1 previous similar message [ 2076.189383] Lustre: lustre-MDT0001: new disk, initializing [ 2076.273966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2076.281890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2078.414393] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2082.948316] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2084.863290] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2087.942970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2088.056863] Lustre: lustre-OST0000: new disk, initializing [ 2088.059406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2088.063503] Lustre: 113607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2088.105435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.108813] Lustre: Skipped 1 previous similar message [ 2090.647288] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2094.245808] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2095.624230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2095.629652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2095.669434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2096.039327] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2096.162261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2098.457497] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2098.567457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2100.705154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2100.705542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.711363] LustreError: Skipped 2 previous similar messages [ 2100.719803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2100.726883] Lustre: Skipped 1 previous similar message [ 2101.621174] Lustre: server umount lustre-OST0000 complete [ 2105.824622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.825924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.831242] Lustre: Skipped 2 previous similar messages [ 2109.877022] Lustre: server umount lustre-MDT0000 complete [ 2111.877066] LustreError: 112644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238195 with bad export cookie 18230965897404544263 [ 2111.879938] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.884427] LustreError: 112644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2112.090839] Lustre: server umount lustre-MDT0001 complete [ 2116.513959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.740824] LustreError: 115670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.749752] LustreError: 115670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2116.781207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.508117] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2122.182757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.207921] LustreError: 115671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.382878] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2126.867749] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2129.190743] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2133.384027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2137.431226] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2140.501138] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2149.919175] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2152.416303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2152.417119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.422891] LustreError: Skipped 1 previous similar message [ 2152.430105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2152.434546] Lustre: Skipped 2 previous similar messages [ 2153.370958] Lustre: server umount lustre-OST0000 complete [ 2157.365153] Lustre: server umount lustre-MDT0000 complete [ 2157.542331] LustreError: 115671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.550292] LustreError: 115671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2159.163390] LustreError: 115650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238242 with bad export cookie 18230965897404545138 [ 2159.167509] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2159.169948] LustreError: 115650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2159.357657] Lustre: server umount lustre-MDT0001 complete [ 2164.425858] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2165.785209] Key type lgssc unregistered [ 2165.951680] LNet: 118938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.956226] LNetError: 118938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.963795] LNet: Removed LNI 192.168.206.107@tcp [ 2166.337249] Key type .llcrypt unregistered [ 2166.338850] Key type ._llcrypt unregistered [ 2174.552382] Key type ._llcrypt registered [ 2174.553947] Key type .llcrypt registered [ 2174.598408] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 2181.991696] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2182.649424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.658626] alg: No test for adler32 (adler32-zlib) [ 2183.536759] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 2183.637644] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2185.231181] Key type lgssc registered [ 2185.733615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2202.031843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2202.043752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.169542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2203.184452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2203.225699] Lustre: lustre-MDT0000: new disk, initializing [ 2203.274617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.289571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2204.952755] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2210.316636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.367876] Lustre: 122031:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2210.396712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2210.399509] Lustre: Skipped 1 previous similar message [ 2210.440199] Lustre: lustre-MDT0001: new disk, initializing [ 2210.485913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2210.503299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2210.510836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2212.204866] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2216.138345] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2217.902300] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2220.997452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2221.121067] Lustre: lustre-OST0000: new disk, initializing [ 2221.123995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2221.127524] Lustre: 123776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2221.167406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.836934] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2225.097564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2225.102383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2225.149810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2227.636704] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2229.344068] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2229.446648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2231.096795] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2231.198980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2235.361199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2235.364060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.365645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2235.366046] LustreError: Skipped 1 previous similar message [ 2235.377611] Lustre: Skipped 1 previous similar message [ 2238.182757] Lustre: server umount lustre-OST0000 complete [ 2240.480402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.481342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2240.489233] Lustre: Skipped 1 previous similar message [ 2240.495068] Lustre: Skipped 2 previous similar messages [ 2245.600195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2245.602969] Lustre: Skipped 1 previous similar message [ 2250.720452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.725363] Lustre: Skipped 1 previous similar message [ 2254.303616] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2254.380554] Lustre: server umount lustre-MDT0000 complete [ 2256.437402] LustreError: 122024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238339 with bad export cookie 8931331993526100983 [ 2256.442611] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.445025] LustreError: 122024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2256.641830] Lustre: server umount lustre-MDT0001 complete [ 2260.670773] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 00:25:44 (1781238344) [ 2263.986975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.241175] LustreError: 126042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.252021] LustreError: 126042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2264.291932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.275650] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2269.664160] LustreError: 126043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.696545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.889685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2272.634161] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2275.119769] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2276.747033] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2279.393159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.395406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2279.399573] Lustre: Skipped 1 previous similar message [ 2279.405402] Lustre: Skipped 1 previous similar message [ 2283.787409] Lustre: server umount lustre-MDT0000 complete [ 2285.536443] LustreError: 126754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.550711] LustreError: 126754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2285.719636] LustreError: 126022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238368 with bad export cookie 8931331993526101865 [ 2285.723244] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2285.726026] LustreError: 126022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2285.876937] Lustre: server umount lustre-MDT0001 complete [ 2290.366932] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 00:26:13 (1781238373) [ 2293.714527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2309.217099] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2314.335642] LustreError: 128227:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 2339.936043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.692060] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2345.207821] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2346.272133] Lustre: server umount lustre-OST0000 complete [ 2349.783062] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 00:27:13 (1781238433) [ 2352.589638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.783732] LustreError: 129557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.367655] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2358.241048] LustreError: 129558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.576793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.788620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2358.791922] Lustre: Skipped 1 previous similar message [ 2360.644460] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2363.182053] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2364.907648] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2368.221827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2370.932890] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2373.304568] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2388.962245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.963338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.969668] Lustre: Skipped 2 previous similar messages [ 2388.976569] Lustre: Skipped 3 previous similar messages [ 2392.811327] Lustre: server umount lustre-MDT0000 complete [ 2394.082215] LustreError: 129557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.088510] LustreError: 129557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2394.318331] LustreError: 129539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238477 with bad export cookie 8931331993526102509 [ 2394.322500] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2394.324034] LustreError: 129539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2394.451223] Lustre: server umount lustre-MDT0001 complete [ 2406.118151] Lustre: server umount lustre-OST0000 complete [ 2409.936673] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 00:28:13 (1781238493) [ 2413.298199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.543493] LustreError: 132887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.588673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.592875] Lustre: Skipped 1 previous similar message [ 2415.290089] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2419.088228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.063508] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2423.566914] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2425.126118] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2428.195794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.883633] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2433.567796] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2435.490391] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2438.635203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2439.712548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2441.461489] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2441.550609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2443.745878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.747216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2443.747656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2443.747675] Lustre: Skipped 1 previous similar message [ 2443.753962] Lustre: Skipped 1 previous similar message [ 2444.451243] Lustre: server umount lustre-OST0000 complete [ 2452.331902] Lustre: server umount lustre-MDT0000 complete [ 2453.985640] LustreError: 132888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.005460] LustreError: 132888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2454.196321] LustreError: 132868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238537 with bad export cookie 8931331993526103566 [ 2454.199329] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.205945] LustreError: 132868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2454.382856] Lustre: server umount lustre-MDT0001 complete [ 2457.656711] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 00:29:01 (1781238541) [ 2460.220567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.679903] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2480.799558] LustreError: 136692:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 2506.399548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.403885] Lustre: Skipped 2 previous similar messages [ 2509.001932] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2511.473162] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2514.731906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.001664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2516.676145] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2520.033202] LustreError: 136718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.042918] LustreError: 136718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2520.215852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.071911] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2524.381960] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2525.989630] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2527.639692] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2527.728714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2529.282566] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2529.373253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2530.784279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2530.788671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.795549] Lustre: Skipped 2 previous similar messages [ 2530.798343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2530.801085] Lustre: Skipped 3 previous similar messages [ 2536.359058] Lustre: server umount lustre-MDT0000 complete [ 2537.810750] LustreError: 136699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238620 with bad export cookie 8931331993526104581 [ 2537.813943] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.815090] LustreError: 136699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2537.934111] Lustre: server umount lustre-MDT0001 complete [ 2549.533354] Lustre: server umount lustre-OST0000 complete [ 2552.652343] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 00:30:36 (1781238636) [ 2555.359415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.847586] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2575.967806] LustreError: 140506:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 2603.993844] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2606.445178] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2609.716199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.056655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2611.875518] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2615.928996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.921919] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2620.264444] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2621.721644] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2624.729367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.827341] Lustre: lustre-OST0001: new disk, initializing [ 2624.830715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2624.834637] Lustre: 143039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2627.432813] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2630.988879] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2631.172389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2631.176605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2631.216682] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2632.568489] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2632.658370] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2634.140139] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2634.216785] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2635.744069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2635.748531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.754014] Lustre: Skipped 2 previous similar messages [ 2635.756541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2635.759199] Lustre: Skipped 3 previous similar messages [ 2641.021465] Lustre: server umount lustre-OST0000 complete [ 2646.500787] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.512215] LustreError: Skipped 1 previous similar message [ 2651.618032] LustreError: 141453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.627357] LustreError: 141453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2657.247137] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2657.304300] Lustre: server umount lustre-OST0001 complete [ 2673.119161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2673.211064] Lustre: server umount lustre-MDT0000 complete [ 2675.144788] LustreError: 140513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238758 with bad export cookie 8931331993526105533 [ 2675.148197] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2675.151537] LustreError: 140513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2675.329730] Lustre: server umount lustre-MDT0001 complete [ 2686.975755] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 00:32:50 (1781238770) [ 2687.797472] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2688.542261] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2689.399719] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 00:32:52 (1781238772) [ 2692.793928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.939690] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2692.954977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2693.043770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.046735] Lustre: Skipped 6 previous similar messages [ 2694.783639] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2698.925996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.988443] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2699.008359] Lustre: 146448:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2700.866702] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2703.450142] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2705.094675] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2708.203072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.313968] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2710.873969] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2713.151324] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2714.137644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2714.914485] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2715.000191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2716.786715] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2716.899433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2719.200561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2719.201771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.205651] LustreError: Skipped 1 previous similar message [ 2719.211029] Lustre: Skipped 5 previous similar messages [ 2719.842196] Lustre: server umount lustre-OST0000 complete [ 2734.733796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.021181] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2738.979868] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.990902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2738.992185] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2739.010958] Lustre: Skipped 1 previous similar message [ 2739.315899] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2741.158818] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2741.250749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2743.025657] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2743.119304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2744.640386] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff964104baf800.ost_server_uuid 50 [ 2745.221110] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff964104baf800.ost_server_uuid in FULL state after 0 sec [ 2745.298865] LustreError: 146456:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.7@tcp) returned error from blocking AST (req@000000009ed8eab8 x1867763696207872 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000e406b1c6/0x7bf27720bd347f17 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.7@tcp remote: 0x92abcbf51d65b867 expref: 5 pid: 147673 timeout: 2845 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2745.317798] LustreError: lustre-MDT0000: A client on nid 192.168.206.7@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2745.323123] LustreError: 146445:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.7@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000e406b1c6/0x7bf27720bd347f17 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.7@tcp remote: 0x92abcbf51d65b867 expref: 6 pid: 147673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2749.410820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2749.415026] LustreError: Skipped 1 previous similar message [ 2752.608957] Lustre: server umount lustre-OST0000 complete [ 2760.426768] Lustre: server umount lustre-MDT0000 complete [ 2762.132708] LustreError: 146442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238845 with bad export cookie 8931331993526106716 [ 2762.135578] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2762.143683] LustreError: 146442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2768.338174] Lustre: server umount lustre-MDT0001 complete [ 2772.945409] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2774.240565] Key type lgssc unregistered [ 2774.402588] LNet: 151915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2774.408115] LNetError: 151915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2774.418372] LNet: Removed LNI 192.168.206.107@tcp [ 2774.769968] Key type .llcrypt unregistered [ 2774.771879] Key type ._llcrypt unregistered [ 2785.715714] Key type ._llcrypt registered [ 2785.717473] Key type .llcrypt registered [ 2785.774833] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2786.235207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2786.245691] alg: No test for adler32 (adler32-zlib) [ 2787.131235] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 2787.221624] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2788.815976] Key type lgssc registered [ 2789.280986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2792.711278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2792.724924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.984411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.626819] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2799.705572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.877337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2801.622312] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2804.193464] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2805.918071] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2809.123144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.294446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.762325] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2814.387898] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2819.560951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2827.000991] Lustre: Failing over lustre-MDT0000 [ 2827.176924] Lustre: server umount lustre-MDT0000 complete [ 2829.795231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.797883] LustreError: 153282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.803777] Lustre: Skipped 2 previous similar messages [ 2829.818345] LustreError: 153282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2834.912764] LustreError: 153287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.920941] LustreError: 153287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2838.257911] LustreError: 153269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.7@tcp arrived at 1781238921 with bad export cookie 6873494668474553458 [ 2838.266222] LustreError: 153269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2840.033708] LustreError: 154406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2840.043027] LustreError: 154406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2841.440912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.508294] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.619967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.658426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2843.460175] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2844.478564] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2844.481449] Lustre: lustre-MDT0000: Denying connection for new client 1cc63872-f9e7-4dfb-b18c-642cd28eb549 (at 192.168.206.7@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2846.692648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2851.809449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.815668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2851.818919] Lustre: Skipped 1 previous similar message [ 2856.929697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.933290] Lustre: Skipped 1 previous similar message [ 2857.556856] Lustre: server umount lustre-OST0000 complete [ 2859.161177] LustreError: 156680:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2859.164748] Lustre: 156048:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2859.169016] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2859.176620] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2859.182112] LustreError: 156048:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001fbe61be x1867764329384832/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2859.190234] LustreError: 156048:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2859.195861] LustreError: 156048:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2859.200598] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 2859.308543] Lustre: server umount lustre-MDT0000 complete [ 2860.844406] LustreError: 153269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238943 with bad export cookie 6873494668474553794 [ 2860.850904] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.853770] LustreError: 153269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2860.863338] LustreError: 152598:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2867.078435] Lustre: server umount lustre-MDT0001 complete [ 2871.373118] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2872.566084] Key type lgssc unregistered [ 2872.714846] LNet: 157417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2872.719126] LNetError: 157417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2872.727433] LNet: Removed LNI 192.168.206.107@tcp [ 2873.066155] Key type .llcrypt unregistered [ 2873.068043] Key type ._llcrypt unregistered [ 2883.313320] Key type ._llcrypt registered [ 2883.315153] Key type .llcrypt registered [ 2883.375955] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2883.849218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2883.889209] alg: No test for adler32 (adler32-zlib) [ 2884.776850] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 2884.881301] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2886.479332] Key type lgssc registered [ 2886.927446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2890.107322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2890.115435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2891.349519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.974050] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2896.481448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2896.649369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2898.186512] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2900.432532] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2901.899843] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2904.459822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.605595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.842108] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2909.133218] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2915.879422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2921.952847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2921.958259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.965047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2926.048761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2926.052504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2926.053828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.064865] Lustre: Skipped 1 previous similar message [ 2927.586721] Lustre: server umount lustre-OST0000 complete [ 2931.168676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.169272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.174128] Lustre: Skipped 1 previous similar message [ 2935.325541] Lustre: server umount lustre-MDT0000 complete [ 2936.288740] LustreError: 158790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.297469] LustreError: 158790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2936.981828] LustreError: 160404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239020 with bad export cookie 3123451256902877453 [ 2936.985754] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.987361] LustreError: 160404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2937.105247] Lustre: server umount lustre-MDT0001 complete [ 2941.487137] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 2942.784173] Key type lgssc unregistered [ 2942.926613] LNet: 161983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.929602] LNetError: 161983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.940765] LNet: Removed LNI 192.168.206.107@tcp [ 2943.320940] Key type .llcrypt unregistered [ 2943.322465] Key type ._llcrypt unregistered [ 2957.434432] Key type ._llcrypt registered [ 2957.435884] Key type .llcrypt registered [ 2957.485468] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 2957.926193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2957.966821] alg: No test for adler32 (adler32-zlib) [ 2958.849597] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 2958.961297] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 2960.559675] Key type lgssc registered [ 2961.078920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2964.693207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2964.703915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.964581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.920116] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2972.625805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.790689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2974.496740] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2976.816929] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2978.528181] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2982.127068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.322642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2985.131036] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 2987.884380] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2992.549231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3002.849934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3002.854670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.857701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.864205] Lustre: Skipped 1 previous similar message [ 3002.872728] Lustre: Skipped 1 previous similar message [ 3003.558642] Lustre: server umount lustre-OST0000 complete [ 3007.968979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.973017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.976160] Lustre: Skipped 1 previous similar message [ 3013.088081] LustreError: 163645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.095987] LustreError: 163645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3013.616307] Lustre: server umount lustre-MDT0000 complete [ 3015.195409] LustreError: 163625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239098 with bad export cookie 11893162271521661031 [ 3015.198419] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3015.201257] LustreError: 163625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3015.325441] Lustre: server umount lustre-MDT0001 complete [ 3019.980460] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3021.235207] Key type lgssc unregistered [ 3021.388600] LNet: 166841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3021.393529] LNetError: 166841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3021.405910] LNet: Removed LNI 192.168.206.107@tcp [ 3021.786414] Key type .llcrypt unregistered [ 3021.788189] Key type ._llcrypt unregistered [ 3031.422458] Key type ._llcrypt registered [ 3031.423936] Key type .llcrypt registered [ 3031.465690] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3031.909777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3031.920714] alg: No test for adler32 (adler32-zlib) [ 3032.788434] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3032.868872] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3034.447196] Key type lgssc registered [ 3034.877130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3039.186498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3039.194058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.378956] Lustre: *** cfs_fail_loc=135, val=0*** [ 3040.420835] LustreError: 168332:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3040.424989] LustreError: 168332:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.107@tcp: cfg command failed: rc = -2 [ 3040.430043] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3040.430043] [ 3040.434962] LustreError: MGC192.168.206.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3040.441160] LustreError: 168293:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3040.445238] LustreError: 168293:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3040.448617] LustreError: 168293:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3040.451413] LustreError: 168293:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3040.496078] Lustre: server umount lustre-MDT0000 complete [ 3040.498050] LustreError: 168293:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3043.253993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3043.465346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.088926] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3048.635216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.813689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3050.476139] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3052.807911] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3054.420211] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3058.146497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.155796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3058.158557] Lustre: Skipped 1 previous similar message [ 3059.169331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3059.171957] Lustre: Skipped 1 previous similar message [ 3061.411695] Lustre: server umount lustre-MDT0000 complete [ 3062.981666] LustreError: 170043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239146 with bad export cookie 14349315792036277078 [ 3062.982576] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.988635] LustreError: 170043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3063.110952] Lustre: server umount lustre-MDT0001 complete [ 3067.349095] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3068.580090] Key type lgssc unregistered [ 3068.736533] LNet: 170777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3068.740721] LNetError: 170777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3068.751075] LNet: Removed LNI 192.168.206.107@tcp [ 3069.089526] Key type .llcrypt unregistered [ 3069.090946] Key type ._llcrypt unregistered [ 3078.955262] Key type ._llcrypt registered [ 3078.956530] Key type .llcrypt registered [ 3079.007378] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3079.146373] Key type .llcrypt unregistered [ 3079.148773] Key type ._llcrypt unregistered [ 3087.635587] Key type ._llcrypt registered [ 3087.637662] Key type .llcrypt registered [ 3087.683876] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3088.084272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3088.121677] alg: No test for adler32 (adler32-zlib) [ 3089.003019] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3089.101724] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3090.703199] Key type lgssc registered [ 3091.210656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3094.674527] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3094.689152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.175314] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3115.295361] LustreError: 172866:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.107@tcp: failed processing log, type 4: rc = -110 [ 3141.919244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.498842] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3146.748088] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3149.758630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3150.029557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.065393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3151.919545] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3155.424466] LustreError: 172891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.432858] LustreError: 172891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3155.756077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.910519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3157.589682] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3159.963289] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3161.624142] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3162.832387] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3165.663870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3165.667051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.674802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3166.176750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3166.180458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.187593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3167.202831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3167.205794] Lustre: Skipped 1 previous similar message [ 3171.489669] Lustre: server umount lustre-OST0000 complete [ 3176.416262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3176.421589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.427970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3179.305296] Lustre: server umount lustre-MDT0000 complete [ 3180.877752] LustreError: 172872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239264 with bad export cookie 8091270930731095918 [ 3180.880756] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3180.883572] LustreError: 172872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3181.036997] Lustre: server umount lustre-MDT0001 complete [ 3185.668275] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3187.043125] Key type lgssc unregistered [ 3187.195511] LNet: 176327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3187.200361] LNetError: 176327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3187.210148] LNet: Removed LNI 192.168.206.107@tcp [ 3187.567747] Key type .llcrypt unregistered [ 3187.569354] Key type ._llcrypt unregistered [ 3198.761342] Key type ._llcrypt registered [ 3198.762791] Key type .llcrypt registered [ 3198.816902] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3199.225319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3199.249249] alg: No test for adler32 (adler32-zlib) [ 3200.143369] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3200.257546] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3201.847156] Key type lgssc registered [ 3202.318479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3205.842395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3205.859490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.092782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.833416] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3212.763290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.956535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3214.944065] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3217.703628] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3219.559904] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3223.022351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3223.213341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.976742] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3228.635570] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3232.745247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3236.549903] Lustre: Failing over lustre-MDT0000 [ 3236.670974] Lustre: server umount lustre-MDT0000 complete [ 3237.857330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.858954] LustreError: 177694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.863094] Lustre: Skipped 1 previous similar message [ 3237.870713] LustreError: 177694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3240.244166] LustreError: 177694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.975956] LustreError: 177700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.981907] LustreError: 177700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3245.347839] LustreError: 177694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.467174] LustreError: 177695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.477542] LustreError: 177695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3250.632959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.702515] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3250.814268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3250.837947] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3252.476620] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3254.558871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3255.589079] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3256.296659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3256.305466] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3256.324693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3261.534448] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3267.044773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3267.052369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.060245] Lustre: Skipped 1 previous similar message [ 3267.064392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3269.600791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3269.605440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.616271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.650399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.653174] Lustre: Skipped 1 previous similar message [ 3272.862534] Lustre: server umount lustre-OST0000 complete [ 3276.769559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3276.770015] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.774462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3276.780347] Lustre: Skipped 1 previous similar message [ 3280.677646] Lustre: server umount lustre-MDT0000 complete [ 3281.892161] LustreError: 177700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.899849] LustreError: 177700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3282.449759] LustreError: 179315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239365 with bad export cookie 6749241898881178252 [ 3282.453097] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.456987] LustreError: 179315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3282.620659] Lustre: server umount lustre-MDT0001 complete [ 3287.166946] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3288.337828] Key type lgssc unregistered [ 3288.477579] LNet: 182249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3288.481228] LNetError: 182249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3288.490328] LNet: Removed LNI 192.168.206.107@tcp [ 3288.811190] Key type .llcrypt unregistered [ 3288.812793] Key type ._llcrypt unregistered [ 3299.939194] Key type ._llcrypt registered [ 3299.940245] Key type .llcrypt registered [ 3299.983657] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3300.419765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.440534] alg: No test for adler32 (adler32-zlib) [ 3301.315427] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3301.420798] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3303.023228] Key type lgssc registered [ 3303.521586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.729739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3306.737037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.929858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.305278] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3312.481184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.629097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3314.135034] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3316.322949] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3317.712881] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3320.221041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3320.364773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3322.482735] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3324.512450] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3333.668950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3336.624070] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3343.281389] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3352.446803] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3358.175980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3358.179814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.185651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.200826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.201694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.202564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.091399] Lustre: server umount lustre-OST0000 complete [ 3368.928355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3368.932205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.941384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.944247] Lustre: Skipped 1 previous similar message [ 3370.611849] Lustre: server umount lustre-MDT0000 complete [ 3372.303320] LustreError: 183602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239455 with bad export cookie 17641293750403991318 [ 3372.303746] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.310672] LustreError: 183602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3372.441864] Lustre: server umount lustre-MDT0001 complete [ 3376.634945] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3377.721890] Key type lgssc unregistered [ 3377.859572] LNet: 187128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3377.864832] LNetError: 187128:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3377.876509] LNet: Removed LNI 192.168.206.107@tcp [ 3378.274499] Key type .llcrypt unregistered [ 3378.276737] Key type ._llcrypt unregistered [ 3386.511423] Key type ._llcrypt registered [ 3386.513141] Key type .llcrypt registered [ 3386.562651] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3386.960396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3386.983685] alg: No test for adler32 (adler32-zlib) [ 3387.849694] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3387.930841] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3389.519211] Key type lgssc registered [ 3389.923701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3392.676045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3392.686602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.857403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.241469] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3398.407266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.584343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3400.027771] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3402.098341] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3403.433834] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3405.959122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.100538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.305069] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3410.537855] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3418.406382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3423.591584] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3432.863417] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3441.160502] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3445.176623] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3454.433608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.435227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.436578] LustreError: Skipped 1 previous similar message [ 3454.442553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3454.446872] Lustre: Skipped 1 previous similar message [ 3459.220397] Lustre: server umount lustre-OST0000 complete [ 3464.672349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.673683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.677948] Lustre: Skipped 2 previous similar messages [ 3466.903753] Lustre: server umount lustre-MDT0000 complete [ 3468.401121] LustreError: 191205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239551 with bad export cookie 6411587630961952309 [ 3468.402248] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3468.406795] LustreError: 191205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3468.533939] Lustre: server umount lustre-MDT0001 complete [ 3472.746665] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3473.943769] Key type lgssc unregistered [ 3474.079551] LNet: 193322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3474.084343] LNetError: 193322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3474.093367] LNet: Removed LNI 192.168.206.107@tcp [ 3474.414389] Key type .llcrypt unregistered [ 3474.415878] Key type ._llcrypt unregistered [ 3484.458271] Key type ._llcrypt registered [ 3484.459414] Key type .llcrypt registered [ 3484.501170] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3484.829402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3484.838094] alg: No test for adler32 (adler32-zlib) [ 3485.708169] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3485.803801] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3487.391162] Key type lgssc registered [ 3487.838350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3491.076557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3491.090100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.294679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.853466] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3497.652339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.838412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3499.541831] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3501.863104] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3503.324337] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3505.830209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3505.975384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.165062] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3510.085242] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3517.222562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3527.651671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.653575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3527.658514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3527.661568] LustreError: Skipped 1 previous similar message [ 3527.664199] Lustre: Skipped 1 previous similar message [ 3530.579131] Lustre: server umount lustre-OST0000 complete [ 3532.768625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.774018] Lustre: Skipped 1 previous similar message [ 3532.776766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.779638] Lustre: Skipped 1 previous similar message [ 3537.888325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.891048] Lustre: Skipped 1 previous similar message [ 3538.084429] Lustre: server umount lustre-MDT0000 complete [ 3539.519265] LustreError: 194675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239622 with bad export cookie 9292820060955604927 [ 3539.522148] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3539.525438] LustreError: 194675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3539.651789] Lustre: server umount lustre-MDT0001 complete [ 3543.661302] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3544.842933] Key type lgssc unregistered [ 3544.986892] LNet: 198036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3544.991090] LNetError: 198036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3545.002598] LNet: Removed LNI 192.168.206.107@tcp [ 3545.333728] Key type .llcrypt unregistered [ 3545.335379] Key type ._llcrypt unregistered [ 3553.543854] Key type ._llcrypt registered [ 3553.545303] Key type .llcrypt registered [ 3553.597893] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3554.108639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3554.143871] alg: No test for adler32 (adler32-zlib) [ 3555.030640] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3555.134627] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3556.727179] Key type lgssc registered [ 3557.163586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3560.310853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3560.319734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3561.517975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.089197] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3566.605625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.787242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3568.513361] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3570.753651] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3572.264213] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3574.777775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.925337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.967324] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3579.131750] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3586.215840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3596.772964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3596.774833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.793263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.795854] Lustre: Skipped 1 previous similar message [ 3598.204938] Lustre: server umount lustre-OST0000 complete [ 3601.891104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3601.892530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3601.900090] Lustre: Skipped 2 previous similar messages [ 3601.906022] Lustre: Skipped 1 previous similar message [ 3602.211048] Lustre: server umount lustre-MDT0000 complete [ 3604.041271] LustreError: 200877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239687 with bad export cookie 2843203012021261481 [ 3604.044653] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.049878] LustreError: 200877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3604.187693] Lustre: server umount lustre-MDT0001 complete [ 3608.727983] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3609.929880] Key type lgssc unregistered [ 3610.080449] LNet: 202602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3610.083731] LNetError: 202602:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3610.093373] LNet: Removed LNI 192.168.206.107@tcp [ 3610.432391] Key type .llcrypt unregistered [ 3610.434060] Key type ._llcrypt unregistered [ 3618.864787] Key type ._llcrypt registered [ 3618.866901] Key type .llcrypt registered [ 3618.918427] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3619.288881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3619.295576] alg: No test for adler32 (adler32-zlib) [ 3620.173409] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3620.282885] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3621.887165] Key type lgssc registered [ 3622.360416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.626462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.634271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.830048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3628.402566] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3632.033042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.195656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.786679] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3636.103474] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3637.684165] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3640.480111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3640.664724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.905323] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3645.115225] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3650.920244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3654.049575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3654.082319] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3654.169129] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3656.419363] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3658.574866] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3659.236747] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3669.884400] Lustre: Permanently deactivating lustre-OST0001 [ 3669.891532] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3675.133938] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3678.908143] Lustre: Permanently reactivating lustre-OST0001 [ 3678.921652] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3678.925294] Lustre: Skipped 2 previous similar messages [ 3681.211491] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.217975] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3681.221663] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3681.227521] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3681.243675] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3689.955850] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.962217] Lustre: Skipped 1 previous similar message [ 3689.967293] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3689.971397] Lustre: Skipped 1 previous similar message [ 3693.607088] Lustre: server umount lustre-OST0001 complete [ 3695.077753] LustreError: 205741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3695.086263] LustreError: 205741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3697.120312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.130257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.139261] Lustre: Skipped 1 previous similar message [ 3697.144402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3699.168754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3699.174962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.182325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3700.195446] LustreError: 205435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3700.204458] LustreError: 205435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3701.786379] Lustre: server umount lustre-OST0000 complete [ 3703.776526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3703.783834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.795016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3703.798058] Lustre: Skipped 2 previous similar messages [ 3709.746991] Lustre: server umount lustre-MDT0000 complete [ 3710.432403] LustreError: 204999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.440723] LustreError: 204999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3711.403493] LustreError: 207001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239794 with bad export cookie 8009406303233642031 [ 3711.405336] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.409392] LustreError: 207001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3711.543963] Lustre: server umount lustre-MDT0001 complete [ 3715.778404] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3717.126238] Key type lgssc unregistered [ 3717.275516] LNet: 208404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3717.280968] LNetError: 208404:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3717.294064] LNet: Removed LNI 192.168.206.107@tcp [ 3717.678389] Key type .llcrypt unregistered [ 3717.680217] Key type ._llcrypt unregistered [ 3734.428771] Key type ._llcrypt registered [ 3734.430368] Key type .llcrypt registered [ 3734.475981] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3734.874335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3734.933269] alg: No test for adler32 (adler32-zlib) [ 3735.807854] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3735.912431] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3737.511377] Key type lgssc registered [ 3738.011876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3741.326375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3741.338486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.488881] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3742.502080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3742.578209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.195263] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3748.110888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.173589] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3748.192264] Lustre: 210566:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3748.341969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3750.215407] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3752.582108] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3754.127603] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3756.871864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3756.977320] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3757.083530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.710090] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3762.227499] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3763.240388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3769.151475] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3777.914036] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3785.712802] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3793.410859] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3805.728897] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3805.732692] Lustre: Skipped 1 previous similar message [ 3829.857513] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3829.861551] Lustre: Skipped 2 previous similar messages [ 3865.211516] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3865.214704] Lustre: Skipped 4 previous similar messages [ 3932.132327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.132334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.133624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.138965] Lustre: Skipped 1 previous similar message [ 3936.600609] Lustre: server umount lustre-OST0000 complete [ 3942.368475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.369218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3942.374960] Lustre: Skipped 1 previous similar message [ 3942.378331] Lustre: Skipped 1 previous similar message [ 3944.685207] Lustre: server umount lustre-MDT0000 complete [ 3946.308527] LustreError: 212198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240029 with bad export cookie 118598411500346771 [ 3946.311294] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3946.313925] LustreError: 212198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3946.477123] Lustre: server umount lustre-MDT0001 complete [ 3951.164903] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 3952.419895] Key type lgssc unregistered [ 3952.561621] LNet: 214993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3952.566671] LNetError: 214993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3952.576400] LNet: Removed LNI 192.168.206.107@tcp [ 3952.917213] Key type .llcrypt unregistered [ 3952.919087] Key type ._llcrypt unregistered [ 3962.840996] Key type ._llcrypt registered [ 3962.842567] Key type .llcrypt registered [ 3962.885073] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 3963.269451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3963.330391] alg: No test for adler32 (adler32-zlib) [ 3964.210730] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 3964.318955] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 3965.919292] Key type lgssc registered [ 3966.391830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3969.755379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3969.766206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.958982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.514569] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3976.025723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.204993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3977.844927] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3980.038933] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3981.444038] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3983.961576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3984.112310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.343723] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_default_debug -1 all [ 3988.559546] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3997.423966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4006.883831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4006.888348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.895890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4007.906682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4007.907468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.907523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4012.953265] Lustre: server umount lustre-OST0000 complete [ 4017.120695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4017.124485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.132262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4017.135514] Lustre: Skipped 1 previous similar message [ 4020.640758] Lustre: server umount lustre-MDT0000 complete [ 4022.266642] LustreError: 216359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240105 with bad export cookie 5030850113144326863 [ 4022.268923] LustreError: MGC192.168.206.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4022.273692] LustreError: 216359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4022.418602] Lustre: server umount lustre-MDT0001 complete [ 4026.678670] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 4028.018539] Key type lgssc unregistered [ 4028.176633] LNet: 219730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4028.182989] LNetError: 219730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4028.196210] LNet: Removed LNI 192.168.206.107@tcp [ 4028.575538] Key type .llcrypt unregistered [ 4028.577463] Key type ._llcrypt unregistered [ 4038.275287] Key type ._llcrypt registered [ 4038.276969] Key type .llcrypt registered [ 4038.325279] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4038.827573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4038.835663] alg: No test for adler32 (adler32-zlib) [ 4039.701783] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 4039.797780] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 4041.383167] Key type lgssc registered [ 4041.808825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4309.447464] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 4310.840366] Key type lgssc unregistered [ 4310.990617] LNet: 221694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4310.995482] LNetError: 221694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4311.006372] LNet: Removed LNI 192.168.206.107@tcp [ 4311.381339] Key type .llcrypt unregistered [ 4311.382956] Key type ._llcrypt unregistered [ 4321.948125] Key type ._llcrypt registered [ 4321.949839] Key type .llcrypt registered [ 4321.995337] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing set_hostid [ 4329.306268] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing load_modules_local [ 4329.686923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4329.701775] alg: No test for adler32 (adler32-zlib) [ 4330.562054] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 4330.653162] LNet: Added LNI 192.168.206.107@tcp [8/256/0/180] [ 4332.239158] Key type lgssc registered [ 4332.675031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4345.148854] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4209 sec ========== 01:00:28 (1781240428) [ 4345.934398] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:00:29 (1781240429) === [ 4347.436354] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:00:30 (1781240430) === [ 4365.160947] Lustre: DEBUG MARKER: oleg607-server.virtnet: executing unload_modules_local [ 4366.334766] Key type lgssc unregistered [ 4366.470691] LNet: 226849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4366.474294] LNetError: 226849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4366.482408] LNet: Removed LNI 192.168.206.107@tcp [ 4366.828639] Key type .llcrypt unregistered [ 4366.830629] Key type ._llcrypt unregistered