[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 545906521 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2592MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002426] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007504] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010006] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012042] Yama: becoming mindful. [ 0.013027] SELinux: Initializing. [ 0.014058] *** VALIDATE selinux *** [ 0.024000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028631] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031123] *** VALIDATE tmpfs *** [ 0.032452] *** VALIDATE proc *** [ 0.033325] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042617] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.044219] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045748] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046035] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051062] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053428] rcu: Hierarchical SRCU implementation. [ 0.055718] smp: Bringing up secondary CPUs ... [ 0.056623] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.062335] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.103129] node 0 deferred pages initialised in 35ms [ 0.106687] devtmpfs: initialized [ 0.107233] x86/mm: Memory block size: 128MB [ 0.109780] gcov: version magic: 0x41383552 [ 0.111086] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.112070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.113270] pinctrl core: initialized pinctrl subsystem [ 0.114207] [ 0.114644] ************************************************************* [ 0.115009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.116010] ** ** [ 0.117009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.118009] ** ** [ 0.119011] ** This means that this kernel is built to expose internal ** [ 0.120010] ** IOMMU data structures, which may compromise security on ** [ 0.121021] ** your system. ** [ 0.122010] ** ** [ 0.123009] ** If you see this message and you are not debugging the ** [ 0.124010] ** kernel, report this immediately to your vendor! ** [ 0.125012] ** ** [ 0.126012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127008] ************************************************************* [ 0.128761] NET: Registered protocol family 16 [ 0.129478] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.130046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.131042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.132806] cpuidle: using governor menu [ 0.133926] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.134427] PCI: Using configuration type 1 for base access [ 0.135377] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.143167] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.145037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.148077] cryptd: max_cpu_qlen set to 1000 [ 0.149373] ACPI: Added _OSI(Module Device) [ 0.151009] ACPI: Added _OSI(Processor Device) [ 0.152009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.154012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.159897] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.166556] ACPI: Interpreter enabled [ 0.168051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.170008] ACPI: Using IOAPIC for interrupt routing [ 0.171148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.176369] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.193446] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.196052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.199016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.203078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.209701] acpiphp: Slot [2] registered [ 0.212114] acpiphp: Slot [5] registered [ 0.213215] acpiphp: Slot [6] registered [ 0.215092] acpiphp: Slot [7] registered [ 0.217106] acpiphp: Slot [8] registered [ 0.219182] acpiphp: Slot [9] registered [ 0.221144] acpiphp: Slot [10] registered [ 0.222158] acpiphp: Slot [3] registered [ 0.224107] acpiphp: Slot [4] registered [ 0.226106] acpiphp: Slot [11] registered [ 0.228103] acpiphp: Slot [12] registered [ 0.230095] acpiphp: Slot [13] registered [ 0.232093] acpiphp: Slot [14] registered [ 0.233140] acpiphp: Slot [15] registered [ 0.235095] acpiphp: Slot [16] registered [ 0.237181] acpiphp: Slot [17] registered [ 0.239159] acpiphp: Slot [18] registered [ 0.241147] acpiphp: Slot [19] registered [ 0.244164] acpiphp: Slot [20] registered [ 0.246105] acpiphp: Slot [21] registered [ 0.248085] acpiphp: Slot [22] registered [ 0.249175] acpiphp: Slot [23] registered [ 0.251119] acpiphp: Slot [24] registered [ 0.253474] acpiphp: Slot [25] registered [ 0.255085] acpiphp: Slot [26] registered [ 0.257076] acpiphp: Slot [27] registered [ 0.259082] acpiphp: Slot [28] registered [ 0.261113] acpiphp: Slot [29] registered [ 0.264106] acpiphp: Slot [30] registered [ 0.266121] acpiphp: Slot [31] registered [ 0.268283] PCI host bridge to bus 0000:00 [ 0.270017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.289341] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293403] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.297447] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.305029] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.309220] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.312016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.314011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.316016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.318427] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322787] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.327762] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.335014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.340015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.350000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.377015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.398014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.423361] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.437013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512968] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.536014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.545015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.570013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.584156] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.593013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.599013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.622014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.634014] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.640013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.651015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.691000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.694000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.712017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.757014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.782027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.821238] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.823454] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.825460] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.828495] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.832275] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.878046] iommu: Default domain type: Passthrough [ 0.879460] SCSI subsystem initialized [ 0.880134] ACPI: bus type USB registered [ 0.881084] usbcore: registered new interface driver usbfs [ 0.882066] usbcore: registered new interface driver hub [ 0.883076] usbcore: registered new device driver usb [ 0.890268] pps_core: LinuxPPS API ver. 1 registered [ 0.892017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.897156] PTP clock support registered [ 0.901133] EDAC MC: Ver: 3.0.0 [ 0.904393] PCI: Using ACPI for IRQ routing [ 0.909718] NetLabel: Initializing [ 0.911028] NetLabel: domain hash size = 128 [ 0.914015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.917102] NetLabel: unlabeled traffic allowed by default [ 0.921101] vgaarb: loaded [ 0.922624] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.925012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.933502] clocksource: Switched to clocksource kvm-clock [ 1.141537] VFS: Disk quotas dquot_6.6.0 [ 1.143312] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.146552] *** VALIDATE ramfs *** [ 1.148251] *** VALIDATE hugetlbfs *** [ 1.150378] pnp: PnP ACPI init [ 1.153458] pnp: PnP ACPI: found 6 devices [ 1.171737] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.175613] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.178181] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.180727] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.183901] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.186973] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.190727] NET: Registered protocol family 2 [ 1.193480] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.199610] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.203618] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.210091] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.215061] TCP: Hash tables configured (established 65536 bind 65536) [ 1.218984] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.223446] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.228414] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.233896] NET: Registered protocol family 1 [ 1.242307] RPC: Registered named UNIX socket transport module. [ 1.246377] RPC: Registered udp transport module. [ 1.250521] RPC: Registered tcp transport module. [ 1.254223] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.258847] NET: Registered protocol family 44 [ 1.263492] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.269997] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.275418] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.281195] PCI: CLS 0 bytes, default 64 [ 1.286214] Unpacking initramfs... [ 4.191859] debug: unmapping init [mem 0xffff9608bcc54000-0xffff9608bffbffff] [ 4.197741] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.201151] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 4.206227] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.244187] Initialise system trusted keyrings [ 5.245629] Key type blacklist registered [ 5.249656] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.258512] zbud: loaded [ 5.261572] *** VALIDATE nfs *** [ 5.262711] *** VALIDATE nfs4 *** [ 5.264281] pstore: using deflate compression [ 5.267982] Platform Keyring initialized [ 5.377380] NET: Registered protocol family 38 [ 5.378916] Key type asymmetric registered [ 5.380452] Asymmetric key parser 'x509' registered [ 5.382474] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.385735] io scheduler mq-deadline registered [ 5.387180] io scheduler kyber registered [ 5.388631] io scheduler bfq registered [ 5.390921] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.394261] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.397248] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.400193] ACPI: Power Button [PWRF] [ 5.406013] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.413875] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.434908] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.449854] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.473179] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.503348] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.533417] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.538434] Non-volatile memory driver v1.3 [ 5.540249] Linux agpgart interface v0.103 [ 5.570599] virtio_blk virtio1: [vda] 136000 512-byte logical blocks (69.6 MB/66.4 MiB) [ 5.573389] vda: detected capacity change from 0 to 69632000 [ 5.596437] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.599858] vdb: detected capacity change from 0 to 1073741824 [ 5.626562] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.629964] vdc: detected capacity change from 0 to 2621440000 [ 5.653824] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.656565] vdd: detected capacity change from 0 to 2621440000 [ 5.682278] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.685336] vde: detected capacity change from 0 to 4294967296 [ 5.706535] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.708971] vdf: detected capacity change from 0 to 4294967296 [ 5.721206] libphy: Fixed MDIO Bus: probed [ 5.726906] usbcore: registered new interface driver usbserial_generic [ 5.729681] usbserial: USB Serial support registered for generic [ 5.733561] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.738890] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.741338] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.743916] mousedev: PS/2 mouse device common for all mice [ 5.747845] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.751844] rtc_cmos 00:05: RTC can wake from S4 [ 5.753453] rtc_cmos 00:05: registered as rtc0 [ 5.753516] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.753551] intel_pstate: CPU model not supported [ 5.763423] hid: raw HID events driver (C) Jiri Kosina [ 5.763605] usbcore: registered new interface driver usbhid [ 5.763608] usbhid: USB HID core driver [ 5.764226] drop_monitor: Initializing network drop monitor service [ 5.779272] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.781639] Initializing XFRM netlink socket [ 5.782597] NET: Registered protocol family 10 [ 5.795556] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.800427] Segment Routing with IPv6 [ 5.802018] NET: Registered protocol family 17 [ 5.805536] mpls_gso: MPLS GSO support [ 5.812787] RAS: Correctable Errors collector initialized. [ 5.815504] AVX version of gcm_enc/dec engaged. [ 5.818399] AES CTR mode by8 optimization enabled [ 5.921812] sched_clock: Marking stable (5921249918, 0)->(7094509355, -1173259437) [ 5.931775] registered taskstats version 1 [ 5.934675] Loading compiled-in X.509 certificates [ 5.937461] zswap: loaded using pool lzo/zbud [ 6.015430] Key type big_key registered [ 6.055507] Key type encrypted registered [ 6.060475] ima: No TPM chip found, activating TPM-bypass! [ 6.085620] ima: Allocated hash algorithm: sha1 [ 6.090657] ima: No architecture policies found [ 6.113379] evm: Initialising EVM extended attributes: [ 6.116452] evm: security.selinux [ 6.118178] evm: security.ima [ 6.120194] evm: security.capability [ 6.122183] evm: HMAC attrs: 0x1 [ 6.126195] rtc_cmos 00:05: setting system clock to 2026-05-29 20:18:36 UTC (1780085916) [ 6.136725] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 6.144596] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 6.149134] Write protecting the kernel read-only data: 28672k [ 6.152512] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 6.155053] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 6.204834] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.222378] systemd[1]: Detected virtualization kvm. [ 6.226397] systemd[1]: Detected architecture x86-64. [ 6.230418] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.279913] systemd[1]: No hostname configured. [ 6.282052] systemd[1]: Set hostname to . [ 6.284270] random: systemd: uninitialized urandom read (16 bytes read) [ 6.287270] systemd[1]: Initializing machine ID from random generator. [ 6.413441] random: ln: uninitialized urandom read (6 bytes read) [ 6.549530] random: systemd: uninitialized urandom read (16 bytes read) [ 6.552499] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.561481] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.571571] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.584572] device-mapper: uevent: version 1.0.3 [ 7.586962] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.041625] virtio_net virtio0 ens2: renamed from eth0 [ 9.067954] random: fast init done [ 9.181407] scsi host0: ata_piix [ 9.208666] scsi host1: ata_piix [ 9.216085] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.284406] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.632468] random: crng init done [ 13.633863] random: 7 urandom warning(s) missed due to ratelimiting [ 15.914358] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK [0[ 18.021580] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) m] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.331941] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.991495] SELinux: Disabled at runtime. [ 21.078561] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.088977] systemd[1]: Detected virtualization kvm. [ 21.092454] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.210746] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.221594] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.236124] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.248401] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.254861] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.266652] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.278516] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel [ 22.359445] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.279640] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.129363] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.343113] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.766849] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.822306] EDAC sbridge: Ver: 1.1.2 [ 28.415994] Key type dns_resolver registered [ 29.167871] NFS: Registering the id_resolver key type [ 29.174166] Key type id_resolver registered [ 29.181058] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 72.751124] libcfs: loading out-of-tree module taints kernel. [ 72.780353] Key type ._llcrypt registered [ 72.782974] Key type .llcrypt registered [ 72.849156] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 82.592217] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 83.396489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.410632] alg: No test for adler32 (adler32-zlib) [ 84.529502] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 84.944098] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 86.592247] Key type lgssc registered [ 87.289789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.675809] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 115.420476] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 121.510568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 121.525429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 122.656252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 122.673456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 122.722922] Lustre: lustre-MDT0000: new disk, initializing [ 122.773597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 122.784950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 124.407449] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.673462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.715051] Lustre: 6485:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 130.731412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 130.733757] Lustre: Skipped 1 previous similar message [ 130.777980] Lustre: lustre-MDT0001: new disk, initializing [ 130.802753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 130.814309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 130.819897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 132.353854] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.048974] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 139.005611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 139.113926] Lustre: lustre-OST0000: new disk, initializing [ 139.117840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 139.121032] Lustre: 8387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 139.149210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.887503] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 142.262330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 142.267595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 142.306515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 148.660876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 148.722891] Lustre: lustre-OST0001: new disk, initializing [ 148.725502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 148.728783] Lustre: 9447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 148.774327] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 150.527647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 150.532228] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 150.558985] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 151.350334] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.287868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 164.406941] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 166.774449] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 168.496246] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 170.252953] Lustre: DEBUG MARKER: Client: 2.17.53.53 [ 171.444662] Lustre: DEBUG MARKER: MDS: 2.17.53.53 [ 172.615187] Lustre: DEBUG MARKER: OSS: 2.17.53.53 [ 173.394492] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri May 29 16:21:24 EDT 2026 [ 182.162767] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 182.892662] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 196.578607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 196.581049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 196.586418] Lustre: Skipped 1 previous similar message [ 201.524789] Lustre: server umount lustre-MDT0000 complete [ 203.551832] LustreError: 7415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086113 with bad export cookie 5821361699558496867 [ 203.554539] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 203.557852] LustreError: 7415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 206.818359] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 206.818988] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 206.826171] Lustre: Skipped 2 previous similar messages [ 206.835207] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 206.838904] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 206.845759] Lustre: Skipped 4 previous similar messages [ 208.866247] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 208.866892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 208.874975] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 208.878351] Lustre: Skipped 1 previous similar message [ 209.734695] Lustre: server umount lustre-MDT0001 complete [ 211.718950] Lustre: server umount lustre-OST0000 complete [ 213.762144] Lustre: server umount lustre-OST0001 complete [ 216.815073] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 221.229323] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 239.581843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.718782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 239.738456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 239.780980] Lustre: lustre-MDT0000: new disk, initializing [ 239.825419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.836171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 241.899415] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 247.263480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.334649] Lustre: 14949:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 247.482768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 249.156870] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 252.975845] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 255.006202] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 258.443899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.591375] Lustre: lustre-OST0000: new disk, initializing [ 258.593892] Lustre: Skipped 1 previous similar message [ 258.596198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 258.599024] Lustre: Skipped 2 previous similar messages [ 258.602791] Lustre: 16690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.648266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.651612] Lustre: Skipped 1 previous similar message [ 260.491151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 260.496867] Lustre: Skipped 1 previous similar message [ 260.500590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 260.538491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 261.228678] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 265.078617] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 266.715167] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 266.807658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 268.441705] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 268.526290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 270.820736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 270.821336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.824928] LustreError: Skipped 1 previous similar message [ 270.830639] Lustre: Skipped 1 previous similar message [ 270.833745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 270.837783] Lustre: Skipped 1 previous similar message [ 275.426636] Lustre: server umount lustre-OST0000 complete [ 278.498382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 278.503717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 278.511468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.327157] Lustre: server umount lustre-MDT0000 complete [ 285.151508] LustreError: 14941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086195 with bad export cookie 5821361699558498400 [ 285.153181] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 285.157662] LustreError: 14941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 285.302329] Lustre: server umount lustre-MDT0001 complete [ 286.934495] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:23:17 (1780086197) [ 289.874810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 290.059108] LustreError: 18803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.092406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 291.650900] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 295.344043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 295.393486] LustreError: 18803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.188617] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 299.639209] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 301.246745] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 304.214475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 306.627911] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 308.922539] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 326.116508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 326.120321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.126788] Lustre: Skipped 1 previous similar message [ 326.130383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 326.134271] Lustre: Skipped 2 previous similar messages [ 329.122785] Lustre: server umount lustre-OST0000 complete [ 331.233925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.238781] LustreError: Skipped 1 previous similar message [ 336.882265] Lustre: server umount lustre-MDT0000 complete [ 337.379168] LustreError: 19513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.386577] LustreError: 19513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 338.673509] LustreError: 18783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086249 with bad export cookie 5821361699558499282 [ 338.676093] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.678710] LustreError: 18783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 338.834705] Lustre: server umount lustre-MDT0001 complete [ 344.325431] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 345.742762] Key type lgssc unregistered [ 345.889558] LNet: 21999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.893348] LNetError: 21999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.902374] LNet: Removed LNI 192.168.203.104@tcp [ 346.394141] Key type .llcrypt unregistered [ 346.395619] Key type ._llcrypt unregistered [ 359.274886] Key type ._llcrypt registered [ 359.277271] Key type .llcrypt registered [ 359.359662] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 359.946371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 359.969261] alg: No test for adler32 (adler32-zlib) [ 360.915906] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 361.057954] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 362.664302] Key type lgssc registered [ 363.251694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 367.791676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 367.811728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 369.083675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 370.842960] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 374.338143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.512039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 376.237825] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 378.759281] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 380.347344] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 383.190554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 383.349666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 385.643934] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 387.947050] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 394.604618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 399.844169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.844430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 399.845675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 399.849138] Lustre: Skipped 1 previous similar message [ 399.852983] LustreError: Skipped 1 previous similar message [ 401.378088] Lustre: server umount lustre-OST0000 complete [ 404.962541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.969687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 404.973860] Lustre: Skipped 2 previous similar messages [ 405.079199] Lustre: server umount lustre-MDT0000 complete [ 406.498742] LustreError: 24090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086316 with bad export cookie 2049110333314943221 [ 406.501722] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 406.504126] LustreError: 24090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 406.626774] Lustre: server umount lustre-MDT0001 complete [ 410.947934] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 412.192434] Key type lgssc unregistered [ 412.346145] LNet: 26987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.350890] LNetError: 26987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 412.362679] LNet: Removed LNI 192.168.203.104@tcp [ 412.729303] Key type .llcrypt unregistered [ 412.730825] Key type ._llcrypt unregistered [ 423.348555] Key type ._llcrypt registered [ 423.350224] Key type .llcrypt registered [ 423.411452] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 423.773880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 423.823103] alg: No test for adler32 (adler32-zlib) [ 424.695549] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 424.791385] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 426.384168] Key type lgssc registered [ 426.896435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 430.252899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 430.264297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.493775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.991348] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 436.254111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.420117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 437.963616] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 440.212768] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 441.682553] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 447.412895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.567323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.856418] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 452.200669] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 455.786599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 466.403267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.406643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.415275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.417628] Lustre: Skipped 1 previous similar message [ 471.521898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.524446] Lustre: Skipped 1 previous similar message [ 472.540885] Lustre: server umount lustre-OST0000 complete [ 476.643153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.645293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.649427] Lustre: Skipped 2 previous similar messages [ 480.165358] Lustre: server umount lustre-MDT0000 complete [ 481.762267] LustreError: 29583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.772395] LustreError: 29583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 481.922110] LustreError: 28340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086392 with bad export cookie 12724253033004087624 [ 481.925168] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 481.930764] LustreError: 28340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 482.078714] Lustre: server umount lustre-MDT0001 complete [ 486.181232] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 487.290892] Key type lgssc unregistered [ 487.434791] LNet: 31896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.438437] LNetError: 31896:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 487.452944] LNet: Removed LNI 192.168.203.104@tcp [ 487.804406] Key type .llcrypt unregistered [ 487.805881] Key type ._llcrypt unregistered [ 497.729518] Key type ._llcrypt registered [ 497.730784] Key type .llcrypt registered [ 497.775833] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 498.197322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 498.215092] alg: No test for adler32 (adler32-zlib) [ 499.084956] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 499.177666] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 500.768212] Key type lgssc registered [ 501.279899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 504.576960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 504.587326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.815430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.455400] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 511.209971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.378119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 512.975815] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 515.210254] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 516.684234] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 519.515494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.666514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.077609] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 524.495655] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 530.923312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 536.035418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 536.037664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.049678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 536.053035] Lustre: Skipped 1 previous similar message [ 539.985221] Lustre: server umount lustre-OST0000 complete [ 542.177046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.179802] LustreError: Skipped 1 previous similar message [ 542.181684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.186489] Lustre: Skipped 1 previous similar message [ 542.188430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.273932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.274071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.558082] Lustre: server umount lustre-MDT0000 complete [ 549.109143] LustreError: 33255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086459 with bad export cookie 10514367528927815344 [ 549.110121] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.114575] LustreError: 33255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 549.251563] Lustre: server umount lustre-MDT0001 complete [ 553.278509] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 554.460219] Key type lgssc unregistered [ 554.608664] LNet: 36446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.613946] LNetError: 36446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 554.622332] LNet: Removed LNI 192.168.203.104@tcp [ 554.974213] Key type .llcrypt unregistered [ 554.976102] Key type ._llcrypt unregistered [ 565.246979] Key type ._llcrypt registered [ 565.248215] Key type .llcrypt registered [ 565.292533] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 565.746257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 565.785865] alg: No test for adler32 (adler32-zlib) [ 566.673591] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 566.781076] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 568.368168] Key type lgssc registered [ 568.796145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 572.062867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 572.070587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.320043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.105218] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 578.933677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.114494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 580.796610] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 583.415236] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 585.140153] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 588.164622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 588.333909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.936369] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 593.368060] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 598.567169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 603.619143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.620450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.621558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.627205] Lustre: Skipped 1 previous similar message [ 608.742510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 608.745519] Lustre: Skipped 3 previous similar messages [ 613.859893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 613.863336] Lustre: Skipped 2 previous similar messages [ 615.392115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 615.457287] Lustre: server umount lustre-OST0000 complete [ 630.240886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.245139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.252444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.256059] Lustre: Skipped 1 previous similar message [ 634.338605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.338798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 634.349370] Lustre: Skipped 1 previous similar message [ 635.490596] Lustre: server umount lustre-MDT0000 complete [ 637.058243] LustreError: 37799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086547 with bad export cookie 1629009576812108532 [ 637.064096] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.267264] Lustre: server umount lustre-MDT0001 complete [ 647.519630] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 648.767161] Key type lgssc unregistered [ 648.907634] LNet: 41165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.911647] LNetError: 41165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.918751] LNet: Removed LNI 192.168.203.104@tcp [ 649.254651] Key type .llcrypt unregistered [ 649.255864] Key type ._llcrypt unregistered [ 659.576348] Key type ._llcrypt registered [ 659.577792] Key type .llcrypt registered [ 659.624771] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 660.050900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 660.093773] alg: No test for adler32 (adler32-zlib) [ 660.969328] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 661.068696] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 662.672218] Key type lgssc registered [ 663.097922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 666.082672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 666.090483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.296867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.863115] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 672.403797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.590467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 674.269202] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 676.627691] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 678.160780] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 681.215533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.402587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.876209] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 686.200121] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 692.648092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 702.041620] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 702.947884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.949377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.953166] Lustre: Skipped 2 previous similar messages [ 702.957744] Lustre: Skipped 2 previous similar messages [ 705.735724] LustreError: 44799:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000f55110dd) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 705.762372] Lustre: server umount lustre-MDT0000 complete [ 707.148853] LustreError: 42535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.156628] LustreError: 42535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 707.269275] LustreError: 44155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086617 with bad export cookie 5360144804554096114 [ 707.272691] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 707.274839] LustreError: 44155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 708.065195] LustreError: 44272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.065413] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.079913] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 712.097206] LustreError: 42536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.097512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 713.458840] Lustre: server umount lustre-MDT0001 complete [ 731.545097] Lustre: server umount lustre-OST0000 complete [ 737.394510] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 738.496167] Key type lgssc unregistered [ 738.635449] LNet: 46061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.638304] LNetError: 46061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.649408] LNet: Removed LNI 192.168.203.104@tcp [ 738.980329] Key type .llcrypt unregistered [ 738.981673] Key type ._llcrypt unregistered [ 749.336557] Key type ._llcrypt registered [ 749.337745] Key type .llcrypt registered [ 749.373591] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 749.688948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 749.728092] alg: No test for adler32 (adler32-zlib) [ 750.593861] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 750.689339] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 752.272161] Key type lgssc registered [ 752.681333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 755.676995] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 755.690372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 771.168326] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 776.288205] LustreError: 47408:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 802.912186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.115699] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 807.207332] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 969.898996] Lustre: server umount lustre-OST0000 complete [ 976.097103] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 977.240986] Key type lgssc unregistered [ 977.387404] LNet: 48952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.390664] LNetError: 48952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.403326] LNet: Removed LNI 192.168.203.104@tcp [ 977.723156] Key type .llcrypt unregistered [ 977.725179] Key type ._llcrypt unregistered [ 987.101305] Key type ._llcrypt registered [ 987.102943] Key type .llcrypt registered [ 987.141110] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 987.520586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 987.557120] alg: No test for adler32 (adler32-zlib) [ 988.411359] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 988.497199] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 990.088335] Key type lgssc registered [ 990.457077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 993.162955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 993.171069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.360350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.850312] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 998.874120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.996933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1000.319205] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1002.196489] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1003.490313] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1005.806715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.919867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.831627] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1009.560941] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1014.243299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.247556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1014.247845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.469476] Lustre: server umount lustre-OST0000 complete [ 1019.363441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1019.363487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.364266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.374260] Lustre: Skipped 1 previous similar message [ 1023.837786] Lustre: server umount lustre-MDT0000 complete [ 1024.480894] LustreError: 50328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.487507] LustreError: 50328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1025.148267] LustreError: 51944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086935 with bad export cookie 5902552698585855505 [ 1025.151293] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1025.152120] LustreError: 51944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1025.273667] Lustre: server umount lustre-MDT0001 complete [ 1029.002508] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1030.026365] Key type lgssc unregistered [ 1030.146429] LNet: 53500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.149257] LNetError: 53500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.157315] LNet: Removed LNI 192.168.203.104@tcp [ 1030.441614] Key type .llcrypt unregistered [ 1030.442697] Key type ._llcrypt unregistered [ 1039.823953] Key type ._llcrypt registered [ 1039.825350] Key type .llcrypt registered [ 1039.864084] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1040.174943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1040.183807] alg: No test for adler32 (adler32-zlib) [ 1041.047069] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1041.135589] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1042.728203] Key type lgssc registered [ 1043.136250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1046.132382] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1046.140456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1061.600306] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1066.720579] LustreError: 54846:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 1093.344170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.723397] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1097.950593] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1100.509181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.693825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.714118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1102.121916] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1105.542132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.709915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1107.289864] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1109.504975] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1111.003779] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1113.059361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.060335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1113.067776] Lustre: Skipped 1 previous similar message [ 1118.179179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1118.181803] Lustre: Skipped 2 previous similar messages [ 1123.301716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1123.306369] Lustre: Skipped 1 previous similar message [ 1126.368373] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1126.431563] Lustre: server umount lustre-OST0000 complete [ 1140.972428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.974648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.977340] Lustre: Skipped 1 previous similar message [ 1140.981688] Lustre: Skipped 1 previous similar message [ 1146.337475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.340481] Lustre: Skipped 1 previous similar message [ 1147.159861] Lustre: server umount lustre-MDT0000 complete [ 1148.511699] LustreError: 55796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087058 with bad export cookie 15926652786537438182 [ 1148.515899] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.518117] LustreError: 55796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1154.695842] Lustre: server umount lustre-MDT0001 complete [ 1158.574428] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1159.632411] Key type lgssc unregistered [ 1159.767473] LNet: 58232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.771089] LNetError: 58232:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.780402] LNet: Removed LNI 192.168.203.104@tcp [ 1160.073472] Key type .llcrypt unregistered [ 1160.075418] Key type ._llcrypt unregistered [ 1170.004562] Key type ._llcrypt registered [ 1170.005719] Key type .llcrypt registered [ 1170.054911] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1170.407603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.453547] alg: No test for adler32 (adler32-zlib) [ 1171.333609] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1171.431292] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1173.032176] Key type lgssc registered [ 1173.457775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1176.409082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1176.419184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.610402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.070118] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1182.163507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.321980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1183.774063] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1185.875692] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1187.224816] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1189.785813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1189.927566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.942351] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1193.920422] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1202.211387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1212.386733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1212.387316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.389094] LustreError: Skipped 1 previous similar message [ 1212.389664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1212.393306] Lustre: Skipped 1 previous similar message [ 1213.523325] Lustre: server umount lustre-OST0000 complete [ 1217.506724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.512917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1217.515046] Lustre: Skipped 2 previous similar messages [ 1222.625055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1222.627279] Lustre: Skipped 1 previous similar message [ 1227.744951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.747138] Lustre: Skipped 1 previous similar message [ 1229.280109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1229.336740] Lustre: server umount lustre-MDT0000 complete [ 1230.645648] LustreError: 59583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087141 with bad export cookie 9703548442824582625 [ 1230.648501] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.650687] LustreError: 59583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1230.775403] Lustre: server umount lustre-MDT0001 complete [ 1234.455200] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1235.533726] Key type lgssc unregistered [ 1235.669317] LNet: 62797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.671994] LNetError: 62797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.680265] LNet: Removed LNI 192.168.203.104@tcp [ 1235.961598] Key type .llcrypt unregistered [ 1235.963084] Key type ._llcrypt unregistered [ 1245.641160] Key type ._llcrypt registered [ 1245.642538] Key type .llcrypt registered [ 1245.680583] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1246.020348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1246.087269] alg: No test for adler32 (adler32-zlib) [ 1246.965104] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1247.054276] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1248.640152] Key type lgssc registered [ 1249.046333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1252.026994] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 16:39:22 (1780087162) [ 1254.381942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1254.389449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.566093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1256.853801] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1259.672928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.791183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.087054] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1262.932296] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1264.210457] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1266.614636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1266.740832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.659225] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1270.487404] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1280.039379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1283.356151] Lustre: Failing over lustre-MDT0000 [ 1283.532830] Lustre: server umount lustre-MDT0000 complete [ 1285.090572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.091621] LustreError: 64454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.096105] Lustre: Skipped 2 previous similar messages [ 1285.103107] LustreError: 64454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1286.231629] LustreError: 64454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.238772] LustreError: 64454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1286.828292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.873639] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1286.941200] Lustre: *** cfs_fail_loc=135, val=0*** [ 1286.986571] LustreError: 67228:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1286.989689] LustreError: 67228:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -2 [ 1286.993717] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1286.993717] [ 1286.997036] LustreError: MGC192.168.203.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1287.002381] LustreError: 67217:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1287.005727] LustreError: 67217:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1287.009149] LustreError: 67217:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1287.011608] LustreError: 67217:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1287.023071] Lustre: server umount lustre-MDT0000 complete [ 1287.024563] LustreError: 67217:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1289.329357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.372079] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1289.458699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.483886] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1290.787789] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1291.339993] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1294.825522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1294.836687] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1294.853422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1299.940166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.940251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1299.945699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1299.950436] Lustre: Skipped 1 previous similar message [ 1301.904182] Lustre: server umount lustre-OST0000 complete [ 1305.057682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1305.057731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.058697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.067359] Lustre: Skipped 2 previous similar messages [ 1309.362773] Lustre: server umount lustre-MDT0000 complete [ 1310.176834] LustreError: 64457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.185827] LustreError: 64457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1310.894480] LustreError: 64440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087221 with bad export cookie 5654820243758208469 [ 1310.901772] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.044273] Lustre: server umount lustre-MDT0001 complete [ 1314.881677] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1316.006585] Key type lgssc unregistered [ 1316.143550] LNet: 68969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.146946] LNetError: 68969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1316.158389] LNet: Removed LNI 192.168.203.104@tcp [ 1316.488655] Key type .llcrypt unregistered [ 1316.490602] Key type ._llcrypt unregistered [ 1325.943399] Key type ._llcrypt registered [ 1325.945086] Key type .llcrypt registered [ 1325.986511] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1326.318460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1326.345032] alg: No test for adler32 (adler32-zlib) [ 1327.222538] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1327.316717] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1328.904157] Key type lgssc registered [ 1329.287460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.883725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.889875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.036757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.262207] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1337.131724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.272141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.611135] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1340.463888] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1341.657483] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1343.947437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1344.053551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.004358] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1347.965995] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1358.375120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1360.691476] Lustre: Failing over lustre-MDT0000 [ 1360.894987] Lustre: server umount lustre-MDT0000 complete [ 1362.912799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1362.915684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.923104] LustreError: 70340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.929794] LustreError: 70340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1363.425464] LustreError: 70341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.426396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.430980] LustreError: 70341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1363.435982] Lustre: Skipped 1 previous similar message [ 1364.155396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.207455] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1364.267056] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1364.324949] LustreError: 73104:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1364.328369] LustreError: 73104:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -9 [ 1364.332347] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1364.332347] [ 1364.336689] LustreError: MGC192.168.203.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1364.342970] LustreError: 73093:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1364.346775] LustreError: 73093:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 1364.350402] LustreError: 73093:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1364.353369] LustreError: 73093:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1364.364909] Lustre: server umount lustre-MDT0000 complete [ 1364.366747] LustreError: 73093:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1366.783351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.827903] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.923028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.959356] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1368.304687] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1368.653961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1372.134044] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1372.143947] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1372.161138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1377.250703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1377.252429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.258681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1377.260468] Lustre: Skipped 1 previous similar message [ 1379.214306] Lustre: server umount lustre-OST0000 complete [ 1382.369566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.370880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.377327] Lustre: Skipped 1 previous similar message [ 1382.383332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.386298] Lustre: Skipped 1 previous similar message [ 1386.649163] Lustre: server umount lustre-MDT0000 complete [ 1387.489708] LustreError: 70337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.499284] LustreError: 70337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1387.958191] LustreError: 70322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087298 with bad export cookie 3862742907399350187 [ 1387.960953] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1387.962968] LustreError: 70322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1388.083232] Lustre: server umount lustre-MDT0001 complete [ 1391.742457] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1392.786545] Key type lgssc unregistered [ 1392.915389] LNet: 74850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.918290] LNetError: 74850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.927296] LNet: Removed LNI 192.168.203.104@tcp [ 1393.262325] Key type .llcrypt unregistered [ 1393.264156] Key type ._llcrypt unregistered [ 1397.523559] Key type ._llcrypt registered [ 1397.525130] Key type .llcrypt registered [ 1397.565347] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1397.673577] Key type .llcrypt unregistered [ 1397.675167] Key type ._llcrypt unregistered [ 1406.797063] Key type ._llcrypt registered [ 1406.798742] Key type .llcrypt registered [ 1406.839350] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1407.289950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1407.322892] alg: No test for adler32 (adler32-zlib) [ 1408.184664] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1408.274255] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1409.864170] Key type lgssc registered [ 1410.236559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1412.884708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1412.890907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.047174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.342808] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1418.107780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.231446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.574304] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1421.451757] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1422.596103] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1424.869147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.990685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.006413] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1428.799934] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1439.338429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1448.929665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.931960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.936966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.954415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.955544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.962385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1453.264361] Lustre: server umount lustre-OST0000 complete [ 1455.073658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.074337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.078658] Lustre: Skipped 1 previous similar message [ 1455.081418] Lustre: Skipped 1 previous similar message [ 1460.193434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.196282] Lustre: Skipped 1 previous similar message [ 1460.689617] Lustre: server umount lustre-MDT0000 complete [ 1461.874341] LustreError: 78307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087372 with bad export cookie 671533763805749639 [ 1461.877028] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.878362] LustreError: 78307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1461.989717] Lustre: server umount lustre-MDT0001 complete [ 1465.521873] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1466.556931] Key type lgssc unregistered [ 1466.678345] LNet: 79887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.681692] LNetError: 79887:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.692394] LNet: Removed LNI 192.168.203.104@tcp [ 1466.984595] Key type .llcrypt unregistered [ 1466.986140] Key type ._llcrypt unregistered [ 1475.872704] Key type ._llcrypt registered [ 1475.874783] Key type .llcrypt registered [ 1475.912301] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1476.436598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.445960] alg: No test for adler32 (adler32-zlib) [ 1477.309863] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1477.395097] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1478.984183] Key type lgssc registered [ 1479.387678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1482.053257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1482.059814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.234806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.548141] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1487.352802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.488791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1488.845805] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1490.816797] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1492.167636] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1494.617039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1494.746366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.711557] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1498.552279] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1508.071223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1512.416981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.420439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.425759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.445576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.446047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.446705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.324925] Lustre: server umount lustre-OST0000 complete [ 1523.681560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.681797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.682562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.682566] Lustre: Skipped 1 previous similar message [ 1523.692846] Lustre: Skipped 1 previous similar message [ 1524.757701] Lustre: server umount lustre-MDT0000 complete [ 1526.022857] LustreError: 82870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087436 with bad export cookie 14095337143962495594 [ 1526.026907] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1526.028470] LustreError: 82870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1526.150794] Lustre: server umount lustre-MDT0001 complete [ 1529.585096] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1530.632240] Key type lgssc unregistered [ 1530.765541] LNet: 84451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.768971] LNetError: 84451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.777534] LNet: Removed LNI 192.168.203.104@tcp [ 1531.088263] Key type .llcrypt unregistered [ 1531.089633] Key type ._llcrypt unregistered [ 1539.623672] Key type ._llcrypt registered [ 1539.624797] Key type .llcrypt registered [ 1539.662127] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1540.017974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.051396] alg: No test for adler32 (adler32-zlib) [ 1540.901059] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1540.984822] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1542.569243] Key type lgssc registered [ 1542.933218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1545.631867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1545.638365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.790294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.044170] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1550.788586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.908933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1552.219936] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1554.147130] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1555.411712] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1557.636284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1557.742229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.588589] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1561.451887] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1572.070458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1581.537327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1581.540356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.544210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1582.562720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.563432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1582.565024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1582.573385] Lustre: Skipped 1 previous similar message [ 1586.128372] Lustre: server umount lustre-OST0000 complete [ 1587.681689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.687850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.690739] Lustre: Skipped 1 previous similar message [ 1592.801296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.803788] Lustre: Skipped 1 previous similar message [ 1593.561229] Lustre: server umount lustre-MDT0000 complete [ 1594.800096] LustreError: 85803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087505 with bad export cookie 15081581887755965639 [ 1594.802895] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1594.804387] LustreError: 85803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1594.916135] Lustre: server umount lustre-MDT0001 complete [ 1598.604397] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1599.689884] Key type lgssc unregistered [ 1599.818538] LNet: 89022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.822512] LNetError: 89022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.833385] LNet: Removed LNI 192.168.203.104@tcp [ 1600.149347] Key type .llcrypt unregistered [ 1600.150944] Key type ._llcrypt unregistered [ 1609.106111] Key type ._llcrypt registered [ 1609.107267] Key type .llcrypt registered [ 1609.143905] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1609.474962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1609.509580] alg: No test for adler32 (adler32-zlib) [ 1610.363275] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1610.442037] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1612.016129] Key type lgssc registered [ 1612.361288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1614.841666] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1614.849989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.304292] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1635.424169] LustreError: 90370:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 1662.048706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.940309] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1665.755829] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1667.663133] Lustre: server umount lustre-OST0000 complete [ 1670.217316] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 16:46:21 (1780087581) [ 1672.374523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.515951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.809197] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1676.592663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.716527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1677.940281] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1679.646270] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1680.842514] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1683.045046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1683.154341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.939632] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1686.666421] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1697.511022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1708.001816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.002472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.003489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1708.005811] Lustre: Skipped 1 previous similar message [ 1708.012035] LustreError: Skipped 1 previous similar message [ 1711.761407] Lustre: server umount lustre-OST0000 complete [ 1713.121590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.122417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.125807] Lustre: Skipped 1 previous similar message [ 1713.129325] Lustre: Skipped 1 previous similar message [ 1718.240863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1718.242660] Lustre: Skipped 1 previous similar message [ 1719.057527] Lustre: server umount lustre-MDT0000 complete [ 1720.195505] LustreError: 91873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087630 with bad export cookie 2052736749228415299 [ 1720.198173] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1720.199581] LustreError: 91873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1720.304994] Lustre: server umount lustre-MDT0001 complete [ 1723.517162] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1724.469063] Key type lgssc unregistered [ 1724.593331] LNet: 95243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.596037] LNetError: 95243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.606258] LNet: Removed LNI 192.168.203.104@tcp [ 1725.869989] Key type .llcrypt unregistered [ 1725.871367] Key type ._llcrypt unregistered [ 1739.543316] Key type ._llcrypt registered [ 1739.544607] Key type .llcrypt registered [ 1739.582338] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1740.037957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1740.052688] alg: No test for adler32 (adler32-zlib) [ 1740.917773] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1741.020166] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1742.608143] Key type lgssc registered [ 1743.005541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1745.691668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1745.698331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.868178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.263914] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1751.089190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.212961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1752.563798] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1754.377131] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1755.688666] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1757.896349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1758.002618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.737398] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1761.413370] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1773.351310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1781.728711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1781.731266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.735757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1783.777936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.779117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1783.779378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.786364] Lustre: Skipped 1 previous similar message [ 1785.742919] Lustre: server umount lustre-OST0000 complete [ 1787.361080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1787.365519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.372797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.110725] Lustre: server umount lustre-MDT0000 complete [ 1794.017419] LustreError: 97386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.022251] LustreError: 97386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1794.361212] LustreError: 98104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087704 with bad export cookie 16321237161451321124 [ 1794.364343] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1794.366552] LustreError: 98104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1794.480395] Lustre: server umount lustre-MDT0001 complete [ 1797.743431] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1798.709282] Key type lgssc unregistered [ 1798.831356] LNet: 100580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.834376] LNetError: 100580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.843436] LNet: Removed LNI 192.168.203.104@tcp [ 1799.144702] Key type .llcrypt unregistered [ 1799.145970] Key type ._llcrypt unregistered [ 1806.073109] Key type ._llcrypt registered [ 1806.074254] Key type .llcrypt registered [ 1806.106096] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1806.434559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.462320] alg: No test for adler32 (adler32-zlib) [ 1807.308986] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1807.384298] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1808.968163] Key type lgssc registered [ 1809.318100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.813300] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.819152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1827.232294] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1832.352162] LustreError: 101782:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 1858.976207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.852335] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1862.702042] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1864.820667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.903911] LustreError: MGC192.168.203.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1864.909592] LustreError: 102690:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1864.913831] LustreError: 102690:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1864.918590] LustreError: 102690:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1864.947572] Lustre: server umount lustre-MDT0000 complete [ 1864.949688] LustreError: 102690:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1876.995702] Lustre: server umount lustre-OST0000 complete [ 1879.934705] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1882.962449] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1894.101618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.183765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1894.195530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1894.230118] Lustre: lustre-MDT0000: new disk, initializing [ 1894.250961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.256887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1895.412501] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1899.298267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.336742] Lustre: 105443:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1899.360895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1899.362506] Lustre: Skipped 1 previous similar message [ 1899.393422] Lustre: lustre-MDT0001: new disk, initializing [ 1899.426690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1899.439831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1899.444169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1900.573206] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1903.537815] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1904.719565] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1906.789990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1906.879186] Lustre: lustre-OST0000: new disk, initializing [ 1906.880778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1906.882843] Lustre: 107185:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1906.909638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.613366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1910.266104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1910.269652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1910.297818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1911.581446] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1912.931294] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1913.003369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1914.227250] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1914.295796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1915.365441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.365577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1915.370050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1915.371033] LustreError: Skipped 1 previous similar message [ 1915.374689] Lustre: Skipped 1 previous similar message [ 1916.944914] Lustre: server umount lustre-OST0000 complete [ 1919.968697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1919.970952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.974276] Lustre: Skipped 1 previous similar message [ 1919.975694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.318382] Lustre: server umount lustre-MDT0000 complete [ 1925.584405] LustreError: 105435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087835 with bad export cookie 8309284511528435812 [ 1925.587852] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.588148] LustreError: 105435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1925.696432] Lustre: server umount lustre-MDT0001 complete [ 1928.462360] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 16:50:39 (1780087839) [ 1929.566807] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1934.858299] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1938.091526] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1950.154411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.246842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1950.258299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1950.298208] Lustre: lustre-MDT0000: new disk, initializing [ 1950.323181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.331172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1951.676037] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1955.811722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.853834] Lustre: 111892:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1955.879656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1955.882375] Lustre: Skipped 1 previous similar message [ 1955.919611] Lustre: lustre-MDT0001: new disk, initializing [ 1955.964790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1955.968762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1957.318232] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1960.519148] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1961.760291] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1964.021253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1964.129550] Lustre: lustre-OST0000: new disk, initializing [ 1964.132299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1964.135655] Lustre: 113632:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1964.172901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.176379] Lustre: Skipped 1 previous similar message [ 1966.084216] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1969.080381] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1970.412318] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1970.680113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1970.683129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1970.715948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1971.499502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1972.725582] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1972.789856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1975.781548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1975.782289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1975.788909] Lustre: Skipped 1 previous similar message [ 1975.791157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1975.794527] Lustre: Skipped 3 previous similar messages [ 1979.541546] Lustre: server umount lustre-OST0000 complete [ 1980.897759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.898636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1980.902958] Lustre: Skipped 2 previous similar messages [ 1986.017188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1986.019090] Lustre: Skipped 2 previous similar messages [ 1995.232152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1995.295150] Lustre: server umount lustre-MDT0000 complete [ 1996.576338] LustreError: 111886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087906 with bad export cookie 8309284511528436680 [ 1996.579513] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1996.582443] LustreError: 111886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1996.708491] Lustre: server umount lustre-MDT0001 complete [ 1999.559180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.761819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.148536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2004.005827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.446027] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2007.202693] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2008.322966] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2010.502336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2010.594340] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2010.596349] Lustre: Skipped 2 previous similar messages [ 2012.454455] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2014.216511] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2017.507322] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2018.786787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2018.787125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.789252] LustreError: Skipped 2 previous similar messages [ 2018.793112] Lustre: Skipped 1 previous similar message [ 2020.560190] Lustre: server umount lustre-OST0000 complete [ 2027.934885] Lustre: server umount lustre-MDT0000 complete [ 2029.025202] LustreError: 115705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.032872] LustreError: 115705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2029.231290] LustreError: 115732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087939 with bad export cookie 8309284511528437548 [ 2029.235138] LustreError: 115732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2029.235393] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.353679] Lustre: server umount lustre-MDT0001 complete [ 2033.038372] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2033.991301] Key type lgssc unregistered [ 2034.113378] LNet: 118973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.117053] LNetError: 118973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.123299] LNet: Removed LNI 192.168.203.104@tcp [ 2034.394213] Key type .llcrypt unregistered [ 2034.395292] Key type ._llcrypt unregistered [ 2040.004549] Key type ._llcrypt registered [ 2040.005778] Key type .llcrypt registered [ 2040.042074] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2046.271740] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2046.667872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2046.675800] alg: No test for adler32 (adler32-zlib) [ 2047.527544] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2047.604216] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2049.184148] Key type lgssc registered [ 2049.533053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2060.367879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2060.373497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.464327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2061.476152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2061.521153] Lustre: lustre-MDT0000: new disk, initializing [ 2061.546095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.555515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2062.854709] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2066.978073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.024381] Lustre: 122062:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2067.046570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2067.048462] Lustre: Skipped 1 previous similar message [ 2067.083787] Lustre: lustre-MDT0001: new disk, initializing [ 2067.114290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.125534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2067.128456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2068.392386] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2071.498972] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2072.811466] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2075.080413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2075.180531] Lustre: lustre-OST0000: new disk, initializing [ 2075.182312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2075.186068] Lustre: 123803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2075.213635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.058338] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2077.942703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2077.945580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2077.977337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2080.081788] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2081.366107] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2081.434518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2082.711099] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2082.781822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2088.419059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.419451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2088.425594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2088.427551] LustreError: Skipped 1 previous similar message [ 2089.485160] Lustre: server umount lustre-OST0000 complete [ 2093.024835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2093.029360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.035362] Lustre: Skipped 1 previous similar message [ 2093.037696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2093.039818] Lustre: Skipped 1 previous similar message [ 2096.789922] Lustre: server umount lustre-MDT0000 complete [ 2098.029587] LustreError: 122055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088008 with bad export cookie 5790592064098202868 [ 2098.033542] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2098.034368] LustreError: 122055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2098.149965] Lustre: server umount lustre-MDT0001 complete [ 2100.827140] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 16:53:31 (1780088011) [ 2102.932360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.093958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.356449] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2107.105813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.230529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.534472] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2110.300739] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2111.496983] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2112.144589] LustreError: 126091:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2112.146487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.147163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.147168] Lustre: Skipped 1 previous similar message [ 2112.149109] LustreError: 126091:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2112.152426] Lustre: Skipped 2 previous similar messages [ 2112.154291] LustreError: 126091:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2117.282144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.286207] Lustre: Skipped 2 previous similar messages [ 2118.290823] Lustre: server umount lustre-MDT0000 complete [ 2119.494370] LustreError: 126044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088029 with bad export cookie 5790592064098203736 [ 2119.497733] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.499232] LustreError: 126044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2119.608924] Lustre: server umount lustre-MDT0001 complete [ 2122.380207] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 16:53:53 (1780088033) [ 2124.539228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2140.000590] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2145.120342] LustreError: 128246:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 2170.720226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.512572] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2174.215606] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2174.916060] Lustre: server umount lustre-OST0000 complete [ 2177.478231] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 16:54:48 (1780088088) [ 2179.634565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.788245] LustreError: 129574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.796746] LustreError: 129574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2181.093061] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2183.857324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.185378] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2186.916587] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2188.124541] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2190.383893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2190.522121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.524447] Lustre: Skipped 2 previous similar messages [ 2192.361798] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2194.114621] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2214.880913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2214.883532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.888602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.426432] Lustre: server umount lustre-MDT0000 complete [ 2218.654386] LustreError: 129555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088129 with bad export cookie 5790592064098204366 [ 2218.657787] LustreError: 129555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2218.658295] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.765390] Lustre: server umount lustre-MDT0001 complete [ 2230.260108] Lustre: server umount lustre-OST0000 complete [ 2233.267714] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 16:55:44 (1780088144) [ 2235.456938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2235.606372] LustreError: 132906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.630634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.930343] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2239.813452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.221346] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2243.073110] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2244.324408] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2246.636270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2248.949947] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2250.687839] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2252.036092] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2260.073545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2260.270739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2261.534913] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2261.602292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2265.570231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.570285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2265.574341] Lustre: Skipped 2 previous similar messages [ 2265.577525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.580796] Lustre: Skipped 4 previous similar messages [ 2268.306183] Lustre: server umount lustre-OST0000 complete [ 2270.690275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2275.672223] Lustre: server umount lustre-MDT0000 complete [ 2275.808989] LustreError: 132906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.814406] LustreError: 132906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2276.872944] LustreError: 132931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088187 with bad export cookie 5790592064098205423 [ 2276.874964] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.876655] LustreError: 132931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2276.986950] Lustre: server umount lustre-MDT0001 complete [ 2279.624904] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 16:56:30 (1780088190) [ 2281.771062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.248582] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2302.368581] LustreError: 136743:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 2327.968403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.970494] Lustre: Skipped 2 previous similar messages [ 2329.818957] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2331.578709] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2333.784842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.917504] LustreError: 136768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.970247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2335.199654] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2337.985042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.350630] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2341.113877] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2342.381026] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2343.716219] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2343.789844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2345.093383] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2345.171353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2348.515545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2348.516714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.522417] Lustre: Skipped 2 previous similar messages [ 2348.525260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.527153] Lustre: Skipped 4 previous similar messages [ 2360.288084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2360.350834] Lustre: server umount lustre-MDT0000 complete [ 2361.532078] LustreError: 136751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088271 with bad export cookie 5790592064098206431 [ 2361.533991] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2361.535281] LustreError: 136751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2361.637816] Lustre: server umount lustre-MDT0001 complete [ 2373.079523] Lustre: server umount lustre-OST0000 complete [ 2375.675162] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 16:58:06 (1780088286) [ 2377.873179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2393.312568] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2398.432378] LustreError: 140559:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 2425.786428] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2427.513691] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2429.700169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.848629] LustreError: 140584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.890734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2431.176421] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2433.921612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.289641] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2436.983518] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2438.126498] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2440.350988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2440.438236] Lustre: lustre-OST0001: new disk, initializing [ 2440.440280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2440.443420] Lustre: 143088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2442.249648] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2443.320080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2443.323062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2443.353355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2445.271174] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2446.823452] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2446.923889] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2448.430460] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2448.526256] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2449.378762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2449.382403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.389157] Lustre: Skipped 2 previous similar messages [ 2449.391713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.393447] Lustre: Skipped 6 previous similar messages [ 2455.294903] Lustre: server umount lustre-OST0000 complete [ 2458.598828] LustreError: 142197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.605676] LustreError: 142197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2462.740030] Lustre: server umount lustre-OST0001 complete [ 2470.120395] Lustre: server umount lustre-MDT0000 complete [ 2471.341027] LustreError: 140566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088381 with bad export cookie 5790592064098207383 [ 2471.344508] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.345275] LustreError: 140566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2471.452149] Lustre: server umount lustre-MDT0001 complete [ 2479.007930] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 16:59:50 (1780088390) [ 2479.559368] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2480.088716] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2480.616810] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 16:59:51 (1780088391) [ 2482.996323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.087734] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2483.096489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2483.167676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.170055] Lustre: Skipped 6 previous similar messages [ 2484.413863] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2487.269570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.303975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2487.314285] Lustre: 146497:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2488.674207] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2490.434264] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2491.598794] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2493.714516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.778450] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2495.530227] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2497.553845] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2498.955315] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2499.091687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2500.075931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2501.517546] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2501.587143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2504.162081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.164094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.164471] LustreError: Skipped 5 previous similar messages [ 2504.167541] Lustre: Skipped 6 previous similar messages [ 2508.299283] Lustre: server umount lustre-OST0000 complete [ 2521.807154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2523.559614] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2523.939658] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.944852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.944880] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2523.952133] Lustre: Skipped 1 previous similar message [ 2525.166542] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2526.494126] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2526.556563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2527.698717] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2527.761451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2528.831451] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95a6031d2000.ost_server_uuid 50 [ 2529.291018] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95a6031d2000.ost_server_uuid in FULL state after 0 sec [ 2529.342800] LustreError: 146503:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.4@tcp) returned error from blocking AST (req@000000006c9583e2 x1866557509781632 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000c96047fc/0x505c50cd6bc50fe3 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.4@tcp remote: 0xe028ab3396c577ad expref: 5 pid: 146502 timeout: 2629 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2529.356674] LustreError: lustre-MDT0000: A client on nid 192.168.203.4@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2529.360414] LustreError: 146492:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.4@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000c96047fc/0x505c50cd6bc50fe3 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.4@tcp remote: 0xe028ab3396c577ad expref: 6 pid: 146502 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2534.371521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.373386] Lustre: Skipped 11 previous similar messages [ 2536.402072] Lustre: server umount lustre-OST0000 complete [ 2543.699276] Lustre: server umount lustre-MDT0000 complete [ 2544.608987] LustreError: 147236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.617707] LustreError: 147236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2544.876436] LustreError: 147220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088455 with bad export cookie 5790592064098208566 [ 2544.879342] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2544.879768] LustreError: 147220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2551.099265] Lustre: server umount lustre-MDT0001 complete [ 2554.497625] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2555.503111] Key type lgssc unregistered [ 2555.631360] LNet: 151988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2555.633529] LNetError: 151988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2555.641346] LNet: Removed LNI 192.168.203.104@tcp [ 2555.922897] Key type .llcrypt unregistered [ 2555.923920] Key type ._llcrypt unregistered [ 2564.759366] Key type ._llcrypt registered [ 2564.761132] Key type .llcrypt registered [ 2564.798772] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2565.066244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2565.073227] alg: No test for adler32 (adler32-zlib) [ 2565.923236] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2566.013551] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2567.592071] Key type lgssc registered [ 2567.896730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2570.215256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2570.222536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.370690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.525320] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2575.150380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.270076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2576.516785] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2578.243251] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2579.348588] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2581.451407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2581.564044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.307101] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2585.057431] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2595.941741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2602.656244] Lustre: Failing over lustre-MDT0000 [ 2602.886872] Lustre: server umount lustre-MDT0000 complete [ 2606.048851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2606.051241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.054895] LustreError: 153361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.062699] LustreError: 153361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2606.562577] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.563163] LustreError: 153356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.568164] Lustre: Skipped 1 previous similar message [ 2611.681136] LustreError: 155517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.686117] LustreError: 155517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2613.779426] LustreError: 153342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.4@tcp arrived at 1780088524 with bad export cookie 12090044113024002015 [ 2613.787371] LustreError: 153342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2616.228399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.269670] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.371456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.390064] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2617.589479] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2618.471222] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2618.473530] Lustre: lustre-MDT0000: Denying connection for new client 555e6730-826e-4bff-8ba1-c402aa0d68c5 (at 192.168.203.4@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2621.417967] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2626.016941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2626.020643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.027067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.531690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2631.316536] Lustre: server umount lustre-OST0000 complete [ 2632.540229] LustreError: 156751:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2632.542552] Lustre: 156117:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2632.545205] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2632.551238] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2632.555105] LustreError: 156117:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007571bfcf x1866558053846656/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2632.560116] LustreError: 156117:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2632.562831] LustreError: 156117:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2632.565750] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2632.664877] Lustre: server umount lustre-MDT0000 complete [ 2633.945158] LustreError: 153340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088544 with bad export cookie 12090044113024002365 [ 2633.953202] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2633.953732] LustreError: 152670:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000027c8e730 x1866558053847552/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2640.137442] Lustre: server umount lustre-MDT0001 complete [ 2643.799442] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2644.774972] Key type lgssc unregistered [ 2644.905338] LNet: 157485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2644.907663] LNetError: 157485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2644.916278] LNet: Removed LNI 192.168.203.104@tcp [ 2645.187722] Key type .llcrypt unregistered [ 2645.188849] Key type ._llcrypt unregistered [ 2654.715139] Key type ._llcrypt registered [ 2654.716276] Key type .llcrypt registered [ 2654.759119] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2655.076761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2655.111996] alg: No test for adler32 (adler32-zlib) [ 2655.968702] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2656.052569] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2657.632090] Key type lgssc registered [ 2657.956284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2660.259849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2660.265026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.407662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.635133] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2665.324236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.446904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2666.708349] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2668.410123] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2669.547400] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2671.590101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.696086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.380989] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2674.993257] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2686.054907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2695.648992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2695.651377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.655775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2696.673803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2696.674211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.674633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2697.807669] Lustre: server umount lustre-OST0000 complete [ 2701.281373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2701.285071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.291365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2701.293280] Lustre: Skipped 1 previous similar message [ 2705.249249] Lustre: server umount lustre-MDT0000 complete [ 2706.499243] LustreError: 158839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088616 with bad export cookie 7955328690120012877 [ 2706.502328] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.503158] LustreError: 158839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2706.620635] Lustre: server umount lustre-MDT0001 complete [ 2710.581149] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2711.660087] Key type lgssc unregistered [ 2711.776366] LNet: 162054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2711.779340] LNetError: 162054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2711.788311] LNet: Removed LNI 192.168.203.104@tcp [ 2712.082993] Key type .llcrypt unregistered [ 2712.086915] Key type ._llcrypt unregistered [ 2723.212887] Key type ._llcrypt registered [ 2723.213875] Key type .llcrypt registered [ 2723.247771] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2723.627259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2723.638627] alg: No test for adler32 (adler32-zlib) [ 2724.509620] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2724.593321] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2726.176155] Key type lgssc registered [ 2726.518509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2729.019258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2729.024911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.171881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.426990] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2734.240767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.362476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2735.609977] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2737.351976] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2738.481865] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2740.744904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2740.855996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.621354] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2744.283580] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2756.199838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2761.699108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.700518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2761.700876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2761.704759] Lustre: Skipped 1 previous similar message [ 2763.088087] Lustre: server umount lustre-OST0000 complete [ 2765.280719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2765.282954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.287165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.289008] Lustre: Skipped 1 previous similar message [ 2766.352820] Lustre: server umount lustre-MDT0000 complete [ 2766.816824] LustreError: 164438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.817557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.822024] LustreError: 164438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2767.513303] LustreError: 165873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088677 with bad export cookie 12472781231811755427 [ 2767.516711] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.519926] LustreError: 165873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2767.636512] Lustre: server umount lustre-MDT0001 complete [ 2771.012390] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2771.930977] Key type lgssc unregistered [ 2772.049370] LNet: 166908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2772.053253] LNetError: 166908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2772.061307] LNet: Removed LNI 192.168.203.104@tcp [ 2772.358130] Key type .llcrypt unregistered [ 2772.359222] Key type ._llcrypt unregistered [ 2780.586423] Key type ._llcrypt registered [ 2780.587682] Key type .llcrypt registered [ 2780.624264] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2780.926679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2780.937127] alg: No test for adler32 (adler32-zlib) [ 2781.789906] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2781.877068] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2783.456134] Key type lgssc registered [ 2783.832881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.268794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2787.275323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.410921] Lustre: *** cfs_fail_loc=135, val=0*** [ 2788.450102] LustreError: 168406:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2788.453380] LustreError: 168406:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -2 [ 2788.457383] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2788.457383] [ 2788.460446] LustreError: MGC192.168.203.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2788.464853] LustreError: 168368:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2788.467278] LustreError: 168368:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2788.469575] LustreError: 168368:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2788.472044] LustreError: 168368:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2788.507614] Lustre: server umount lustre-MDT0000 complete [ 2788.508953] LustreError: 168368:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2790.628248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.773986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.000215] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2794.639040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.760684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2795.983971] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2797.649903] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2798.874633] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2799.548397] LustreError: 168806:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2799.551752] LustreError: 168806:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2799.554397] LustreError: 168806:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2800.097560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2800.098035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.104505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.106215] Lustre: Skipped 1 previous similar message [ 2805.217394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2805.221855] Lustre: Skipped 1 previous similar message [ 2805.714855] Lustre: server umount lustre-MDT0000 complete [ 2806.875102] LustreError: 170116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088717 with bad export cookie 6344410049423622969 [ 2806.878048] LustreError: 170116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2806.878849] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.982954] Lustre: server umount lustre-MDT0001 complete [ 2810.075168] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2810.981555] Key type lgssc unregistered [ 2811.100294] LNet: 170851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2811.102654] LNetError: 170851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2811.110250] LNet: Removed LNI 192.168.203.104@tcp [ 2811.361685] Key type .llcrypt unregistered [ 2811.362733] Key type ._llcrypt unregistered [ 2818.094072] Key type ._llcrypt registered [ 2818.095109] Key type .llcrypt registered [ 2818.132665] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2818.231335] Key type .llcrypt unregistered [ 2818.232368] Key type ._llcrypt unregistered [ 2824.845768] Key type ._llcrypt registered [ 2824.846876] Key type .llcrypt registered [ 2824.886293] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2825.251986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2825.311742] alg: No test for adler32 (adler32-zlib) [ 2826.169238] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2826.261207] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2827.848148] Key type lgssc registered [ 2828.182836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2830.672910] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2830.678286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2846.112241] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2851.232216] LustreError: 172926:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 2877.856234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.707973] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2881.414102] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2883.564613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.736160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2883.759563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2884.944332] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2887.600247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.720487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.866731] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2890.526852] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2891.660599] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2892.676036] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2901.987221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.988500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2901.991703] Lustre: Skipped 1 previous similar message [ 2907.107113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2907.110232] Lustre: Skipped 2 previous similar messages [ 2912.225460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.272110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2914.318050] Lustre: server umount lustre-OST0000 complete [ 2917.345733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.346591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.350464] Lustre: Skipped 1 previous similar message [ 2917.355009] Lustre: Skipped 2 previous similar messages [ 2921.622877] Lustre: server umount lustre-MDT0000 complete [ 2922.464920] LustreError: 173853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.469445] LustreError: 173853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2922.823085] LustreError: 172932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088833 with bad export cookie 5931534196662739090 [ 2922.826539] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.827017] LustreError: 172932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2922.941426] Lustre: server umount lustre-MDT0001 complete [ 2926.376216] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2927.375629] Key type lgssc unregistered [ 2927.499361] LNet: 176577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2927.501607] LNetError: 176577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2927.512849] LNet: Removed LNI 192.168.203.104@tcp [ 2927.802771] Key type .llcrypt unregistered [ 2927.803772] Key type ._llcrypt unregistered [ 2936.460482] Key type ._llcrypt registered [ 2936.462130] Key type .llcrypt registered [ 2936.500681] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2936.834534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2936.891315] alg: No test for adler32 (adler32-zlib) [ 2937.756099] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2937.840752] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2939.424202] Key type lgssc registered [ 2939.836101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2943.017555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2943.026485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.191482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2945.609838] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2948.393876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.516253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2949.762333] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2951.529101] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2952.729760] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2955.085576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2955.209900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.130417] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2958.966159] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2969.581638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2970.910136] Lustre: Failing over lustre-MDT0000 [ 2971.017075] Lustre: server umount lustre-MDT0000 complete [ 2974.176926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2974.181029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.187698] LustreError: 177948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.192967] LustreError: 177948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2974.690380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.691095] LustreError: 180101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.694297] Lustre: Skipped 1 previous similar message [ 2979.810090] LustreError: 179308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.821328] LustreError: 179308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2984.027050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.070406] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2984.167836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.203605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2985.036597] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2985.494025] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2987.000584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2989.547584] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2989.559399] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2989.590890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2995.983125] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3001.826510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3001.829967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.835889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.385344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.391420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.401918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3006.684930] Lustre: server umount lustre-OST0000 complete [ 3010.017901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3010.018710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.019254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.019258] Lustre: Skipped 1 previous similar message [ 3010.028034] Lustre: Skipped 1 previous similar message [ 3015.138141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.143372] Lustre: Skipped 2 previous similar messages [ 3020.258868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.262929] Lustre: Skipped 1 previous similar message [ 3022.304168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3022.379872] Lustre: server umount lustre-MDT0000 complete [ 3023.984173] LustreError: 177928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088934 with bad export cookie 11449989164003386409 [ 3023.991052] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.123411] Lustre: server umount lustre-MDT0001 complete [ 3028.025885] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3029.152183] Key type lgssc unregistered [ 3029.284324] LNet: 182621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3029.286697] LNetError: 182621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3029.294320] LNet: Removed LNI 192.168.203.104@tcp [ 3029.622841] Key type .llcrypt unregistered [ 3029.624053] Key type ._llcrypt unregistered [ 3039.387174] Key type ._llcrypt registered [ 3039.388429] Key type .llcrypt registered [ 3039.447254] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3039.789412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3039.832762] alg: No test for adler32 (adler32-zlib) [ 3040.709594] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3040.787930] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3042.360142] Key type lgssc registered [ 3042.728142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3045.655184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3045.660916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.808866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.975863] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3050.918153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.091148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3052.501293] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3054.357004] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3055.554584] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3058.664333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3058.860564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3061.432347] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3063.968185] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3072.174331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3075.513176] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3077.037234] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3086.354094] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3091.938918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3091.944679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.955121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.963311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.966041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.966592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.980966] Lustre: Skipped 1 previous similar message [ 3097.898792] Lustre: server umount lustre-OST0000 complete [ 3102.177510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3102.182094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3102.190357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3105.751568] Lustre: server umount lustre-MDT0000 complete [ 3106.878094] LustreError: 186461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089017 with bad export cookie 9722923761998503318 [ 3106.880504] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.883022] LustreError: 186461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3106.992652] Lustre: server umount lustre-MDT0001 complete [ 3110.333964] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3111.245456] Key type lgssc unregistered [ 3111.377372] LNet: 187498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3111.379439] LNetError: 187498:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3111.390299] LNet: Removed LNI 192.168.203.104@tcp [ 3111.659371] Key type .llcrypt unregistered [ 3111.660410] Key type ._llcrypt unregistered [ 3118.731587] Key type ._llcrypt registered [ 3118.732464] Key type .llcrypt registered [ 3118.766582] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3119.299565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3119.305590] alg: No test for adler32 (adler32-zlib) [ 3120.158089] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3120.240489] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3121.816291] Key type lgssc registered [ 3122.305578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.146845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.158037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.371640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.844822] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3130.895436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.039417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3132.582986] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3134.286595] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3135.889131] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3138.684766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3138.796935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.226624] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3143.793686] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3151.083360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3156.332272] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3164.468281] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3173.563468] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3179.205414] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3187.175418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3187.175994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.180875] LustreError: Skipped 1 previous similar message [ 3187.190362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3187.196925] Lustre: Skipped 1 previous similar message [ 3192.291212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.297074] Lustre: Skipped 1 previous similar message [ 3192.357118] Lustre: server umount lustre-OST0000 complete [ 3197.411216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.412810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3197.419515] Lustre: Skipped 2 previous similar messages [ 3197.429889] Lustre: Skipped 1 previous similar message [ 3200.081655] Lustre: server umount lustre-MDT0000 complete [ 3201.634938] LustreError: 191086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089112 with bad export cookie 18016274424595406570 [ 3201.640934] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.646551] LustreError: 191086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3201.789390] Lustre: server umount lustre-MDT0001 complete [ 3206.218707] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3207.495739] Key type lgssc unregistered [ 3207.639366] LNet: 193647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3207.643322] LNetError: 193647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3207.653311] LNet: Removed LNI 192.168.203.104@tcp [ 3207.999141] Key type .llcrypt unregistered [ 3208.000794] Key type ._llcrypt unregistered [ 3217.204545] Key type ._llcrypt registered [ 3217.206179] Key type .llcrypt registered [ 3217.250280] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3217.638308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3217.659437] alg: No test for adler32 (adler32-zlib) [ 3218.551989] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3218.662889] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3220.288182] Key type lgssc registered [ 3220.670643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3224.724671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3224.737889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3225.963651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.261648] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3230.566783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.685737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3232.145730] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3234.251966] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3235.530545] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3238.287704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3238.410060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.596903] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3242.480577] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3250.728098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3260.898790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3260.903690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.905076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3260.909429] Lustre: Skipped 1 previous similar message [ 3260.915445] Lustre: Skipped 1 previous similar message [ 3263.322854] Lustre: server umount lustre-OST0000 complete [ 3266.531782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3266.545975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.554806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3271.137768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.138575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3271.158762] Lustre: Skipped 1 previous similar message [ 3272.852855] Lustre: server umount lustre-MDT0000 complete [ 3275.815967] LustreError: 197194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089186 with bad export cookie 9533461805642424860 [ 3275.816931] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3275.828085] LustreError: 197194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3276.125831] Lustre: server umount lustre-MDT0001 complete [ 3286.413850] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3288.675400] Key type lgssc unregistered [ 3288.902563] LNet: 198376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3288.908437] LNetError: 198376:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3289.960413] LNet: Removed LNI 192.168.203.104@tcp [ 3290.396718] Key type .llcrypt unregistered [ 3290.398834] Key type ._llcrypt unregistered [ 3299.946097] Key type ._llcrypt registered [ 3299.947973] Key type .llcrypt registered [ 3300.002046] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3300.436955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.484871] alg: No test for adler32 (adler32-zlib) [ 3301.360171] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3301.467862] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3303.065795] Key type lgssc registered [ 3303.561405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3307.006694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3307.017272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.241491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.261380] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3313.828694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.004334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3315.700700] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3318.078362] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3319.519141] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3321.929895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3322.078685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.223303] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3326.412558] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3333.350736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3343.843933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.845638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.850092] Lustre: Skipped 1 previous similar message [ 3343.850420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3343.853067] Lustre: Skipped 1 previous similar message [ 3347.855851] Lustre: server umount lustre-OST0000 complete [ 3349.985301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3349.989477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.996412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3354.081500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3354.081710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.355752] Lustre: server umount lustre-MDT0000 complete [ 3356.777035] LustreError: 199583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089267 with bad export cookie 3766129383470487145 [ 3356.780484] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3356.782062] LustreError: 199583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3356.917780] Lustre: server umount lustre-MDT0001 complete [ 3360.854808] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3361.942732] Key type lgssc unregistered [ 3362.082457] LNet: 202947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3362.085761] LNetError: 202947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3362.099561] LNet: Removed LNI 192.168.203.104@tcp [ 3362.404281] Key type .llcrypt unregistered [ 3362.405842] Key type ._llcrypt unregistered [ 3370.367181] Key type ._llcrypt registered [ 3370.368387] Key type .llcrypt registered [ 3370.420273] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3370.764029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3370.822046] alg: No test for adler32 (adler32-zlib) [ 3371.698153] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3371.801274] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3373.400221] Key type lgssc registered [ 3373.888690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3376.872021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3376.882234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.068483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.391777] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3382.598765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.732468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3384.063377] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3386.028571] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3387.405163] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3390.023977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.198835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.288456] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3394.339475] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3403.496936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3407.431476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3407.469068] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3407.561628] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3409.580804] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3411.535384] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3412.219730] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3422.618206] Lustre: Permanently deactivating lustre-OST0001 [ 3422.624441] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3432.440284] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3435.887620] Lustre: Permanently reactivating lustre-OST0001 [ 3435.894555] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3435.896988] Lustre: Skipped 2 previous similar messages [ 3439.160609] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.166291] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3439.169587] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3439.173913] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3439.187021] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3449.313184] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.315958] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.320084] Lustre: Skipped 1 previous similar message [ 3449.321952] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3449.825724] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.829561] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3450.851367] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3450.854124] Lustre: Skipped 1 previous similar message [ 3453.508446] Lustre: server umount lustre-OST0001 complete [ 3454.945092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.945115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.947802] LustreError: Skipped 1 previous similar message [ 3454.952465] Lustre: Skipped 1 previous similar message [ 3454.955265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3454.959298] Lustre: Skipped 1 previous similar message [ 3455.972317] LustreError: 205768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.978302] LustreError: 205768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3461.091274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3461.092430] LustreError: 205767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.093425] Lustre: Skipped 3 previous similar messages [ 3461.100888] LustreError: 205767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3466.211934] LustreError: 205767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.218646] LustreError: 205767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3469.280158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3469.334073] Lustre: server umount lustre-OST0000 complete [ 3471.331048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.331731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.335483] Lustre: Skipped 1 previous similar message [ 3471.337647] Lustre: Skipped 2 previous similar messages [ 3476.772634] Lustre: server umount lustre-MDT0000 complete [ 3478.095792] LustreError: 204155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089388 with bad export cookie 3809229737190628141 [ 3478.100842] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.101847] LustreError: 204155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3478.229492] Lustre: server umount lustre-MDT0001 complete [ 3482.189693] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3483.326598] Key type lgssc unregistered [ 3483.454359] LNet: 208818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3483.458296] LNetError: 208818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3483.469338] LNet: Removed LNI 192.168.203.104@tcp [ 3483.768420] Key type .llcrypt unregistered [ 3483.769493] Key type ._llcrypt unregistered [ 3498.948225] Key type ._llcrypt registered [ 3498.949993] Key type .llcrypt registered [ 3498.998941] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3499.439258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.449628] alg: No test for adler32 (adler32-zlib) [ 3500.314625] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3500.410793] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3501.992174] Key type lgssc registered [ 3502.404909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3505.259455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3505.266209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.365446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3506.375872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3506.443262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.845859] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3510.835244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.879127] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3510.892524] Lustre: 210980:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3511.006545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3512.609342] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3514.697609] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3516.064059] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3518.530391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3518.605332] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3518.675192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.653429] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3520.661493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3522.614089] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3524.210772] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3531.826162] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3538.391154] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3545.968569] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3559.143478] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3559.146574] Lustre: Skipped 1 previous similar message [ 3582.062951] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3582.066347] Lustre: Skipped 2 previous similar messages [ 3619.014415] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3619.017237] Lustre: Skipped 4 previous similar messages [ 3684.837571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.839249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3684.839864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3684.844529] Lustre: Skipped 1 previous similar message [ 3687.130173] Lustre: server umount lustre-OST0000 complete [ 3689.953452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.957976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3689.959665] Lustre: Skipped 2 previous similar messages [ 3694.563995] Lustre: server umount lustre-MDT0000 complete [ 3696.013218] LustreError: 213205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089606 with bad export cookie 7220992896383337829 [ 3696.016515] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3696.018126] LustreError: 213205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3696.148578] Lustre: server umount lustre-MDT0001 complete [ 3700.177256] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3701.330477] Key type lgssc unregistered [ 3701.474603] LNet: 215425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3701.480031] LNetError: 215425:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3701.489938] LNet: Removed LNI 192.168.203.104@tcp [ 3701.834622] Key type .llcrypt unregistered [ 3701.836816] Key type ._llcrypt unregistered [ 3711.665780] Key type ._llcrypt registered [ 3711.667334] Key type .llcrypt registered [ 3711.710968] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3712.051815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3712.086087] alg: No test for adler32 (adler32-zlib) [ 3712.943629] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3713.024925] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3714.608112] Key type lgssc registered [ 3714.986254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3717.685606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3717.693332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.867338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.184735] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3723.084307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.220628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3724.555333] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3726.409954] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3727.729028] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3730.242527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3730.377853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3732.328821] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3734.190587] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3743.718695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3753.954089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3753.955068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.956552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.956672] LustreError: Skipped 1 previous similar message [ 3753.960930] Lustre: Skipped 1 previous similar message [ 3756.884358] Lustre: server umount lustre-OST0000 complete [ 3759.073602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3759.074024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.081576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.084275] Lustre: Skipped 2 previous similar messages [ 3764.193214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.196585] Lustre: Skipped 1 previous similar message [ 3764.315207] Lustre: server umount lustre-MDT0000 complete [ 3765.668388] LustreError: 216776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089676 with bad export cookie 14638042785689525245 [ 3765.672588] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3765.675067] LustreError: 216776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3765.792341] Lustre: server umount lustre-MDT0001 complete [ 3769.604791] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3770.724806] Key type lgssc unregistered [ 3770.869710] LNet: 220149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3770.873707] LNetError: 220149:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3770.885547] LNet: Removed LNI 192.168.203.104@tcp [ 3771.202288] Key type .llcrypt unregistered [ 3771.204071] Key type ._llcrypt unregistered [ 3780.936764] Key type ._llcrypt registered [ 3780.938382] Key type .llcrypt registered [ 3780.988835] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3781.347467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3781.436432] alg: No test for adler32 (adler32-zlib) [ 3782.315676] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3782.414131] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3784.008322] Key type lgssc registered [ 3784.465770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4047.694758] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4048.640787] Key type lgssc unregistered [ 4048.763325] LNet: 222119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4048.766281] LNetError: 222119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4048.779279] LNet: Removed LNI 192.168.203.104@tcp [ 4049.056606] Key type .llcrypt unregistered [ 4049.057649] Key type ._llcrypt unregistered [ 4056.669849] Key type ._llcrypt registered [ 4056.671274] Key type .llcrypt registered [ 4056.703523] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4062.603409] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4062.929232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4062.946844] alg: No test for adler32 (adler32-zlib) [ 4063.787537] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 4063.857637] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4065.432134] Key type lgssc registered [ 4065.821365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4075.549875] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3902 sec ========== 17:26:26 (1780089986) [ 4076.106607] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:26:27 (1780089987) === [ 4077.232817] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:26:28 (1780089988) === [ 4089.904927] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4090.847414] Key type lgssc unregistered [ 4090.967338] LNet: 227273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4090.969767] LNetError: 227273:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4090.978271] LNet: Removed LNI 192.168.203.104@tcp [ 4091.243472] Key type .llcrypt unregistered [ 4091.244716] Key type ._llcrypt unregistered