[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 615568311 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003054] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004000] kvm-guest: setup PV IPIs [ 0.006442] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008030] pid_max: default: 32768 minimum: 301 [ 0.009000] LSM: Security Framework initializing [ 0.010049] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.013052] *** VALIDATE selinux *** [ 0.026255] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.034567] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.036146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037309] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039193] *** VALIDATE tmpfs *** [ 0.041079] *** VALIDATE proc *** [ 0.043099] *** VALIDATE cgroup *** [ 0.044006] *** VALIDATE cgroup2 *** [ 0.045454] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.046149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.047000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.047037] Spectre V2 : User space: Vulnerable [ 0.048012] Speculative Store Bypass: Vulnerable [ 0.051405] debug: unmapping init [mem 0xffffffffa8259000-0xffffffffa8260fff] [ 0.054428] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.055790] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.056019] ... version: 2 [ 0.057008] ... bit width: 48 [ 0.058015] ... generic registers: 4 [ 0.059017] ... value mask: 0000ffffffffffff [ 0.060037] ... max period: 00007fffffffffff [ 0.061013] ... fixed-purpose events: 3 [ 0.062015] ... event mask: 000000070000000f [ 0.063634] rcu: Hierarchical SRCU implementation. [ 0.065997] smp: Bringing up secondary CPUs ... [ 0.068030] x86: Booting SMP configuration: [ 0.069511] .... node #0, CPUs: #1 #2 #3 [ 0.081103] smp: Brought up 1 node, 4 CPUs [ 0.083017] smpboot: Max logical packages: 1 [ 0.084015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133085] node 0 deferred pages initialised in 41ms [ 0.141160] devtmpfs: initialized [ 0.142290] x86/mm: Memory block size: 128MB [ 0.144000] gcov: version magic: 0x41383552 [ 0.148728] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150333] pinctrl core: initialized pinctrl subsystem [ 0.151126] [ 0.151470] ************************************************************* [ 0.152008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153006] ** ** [ 0.154012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155009] ** ** [ 0.156008] ** This means that this kernel is built to expose internal ** [ 0.157018] ** IOMMU data structures, which may compromise security on ** [ 0.158009] ** your system. ** [ 0.159018] ** ** [ 0.160016] ** If you see this message and you are not debugging the ** [ 0.161019] ** kernel, report this immediately to your vendor! ** [ 0.162016] ** ** [ 0.164014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165013] ************************************************************* [ 0.167000] NET: Registered protocol family 16 [ 0.167000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.184128] cpuidle: using governor menu [ 0.185627] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186536] PCI: Using configuration type 1 for base access [ 0.187000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194196] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197075] cryptd: max_cpu_qlen set to 1000 [ 0.199316] ACPI: Added _OSI(Module Device) [ 0.200018] ACPI: Added _OSI(Processor Device) [ 0.202012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214726] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229651] ACPI: Interpreter enabled [ 0.233098] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237015] ACPI: Using IOAPIC for interrupt routing [ 0.241120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276594] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.290026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.295101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.303000] acpiphp: Slot [2] registered [ 0.303000] acpiphp: Slot [3] registered [ 0.304125] acpiphp: Slot [4] registered [ 0.308060] acpiphp: Slot [5] registered [ 0.310135] acpiphp: Slot [6] registered [ 0.313018] acpiphp: Slot [7] registered [ 0.315130] acpiphp: Slot [8] registered [ 0.319142] acpiphp: Slot [9] registered [ 0.322108] acpiphp: Slot [10] registered [ 0.326101] acpiphp: Slot [11] registered [ 0.329151] acpiphp: Slot [12] registered [ 0.332098] acpiphp: Slot [13] registered [ 0.334210] acpiphp: Slot [14] registered [ 0.336079] acpiphp: Slot [15] registered [ 0.340027] acpiphp: Slot [16] registered [ 0.342153] acpiphp: Slot [17] registered [ 0.346122] acpiphp: Slot [18] registered [ 0.349172] acpiphp: Slot [19] registered [ 0.353038] acpiphp: Slot [20] registered [ 0.356193] acpiphp: Slot [21] registered [ 0.359095] acpiphp: Slot [22] registered [ 0.362097] acpiphp: Slot [23] registered [ 0.365048] acpiphp: Slot [24] registered [ 0.367153] acpiphp: Slot [25] registered [ 0.370165] acpiphp: Slot [26] registered [ 0.372136] acpiphp: Slot [27] registered [ 0.375178] acpiphp: Slot [28] registered [ 0.379148] acpiphp: Slot [29] registered [ 0.383030] acpiphp: Slot [30] registered [ 0.384112] acpiphp: Slot [31] registered [ 0.388112] PCI host bridge to bus 0000:00 [ 0.392025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.399030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.402031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.408046] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.415036] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.419579] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.421192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.425638] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.430986] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.448019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.458125] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.463020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.468019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.472018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.479506] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.485439] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.490044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.496079] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 10742 usecs [ 0.500625] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.511018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.542021] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.548021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.557000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.568019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.577019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.604018] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.619190] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.628016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.637017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.659017] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.669705] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.681018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.692017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.715017] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.732722] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.746018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.756024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.766015] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.773462] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.789020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.795014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.820022] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.840814] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.851014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.866019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.884018] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.893060] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.895345] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.897338] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.900313] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.902174] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.906139] iommu: Default domain type: Passthrough [ 0.908363] SCSI subsystem initialized [ 0.909110] ACPI: bus type USB registered [ 0.910072] usbcore: registered new interface driver usbfs [ 0.916118] usbcore: registered new interface driver hub [ 0.926149] usbcore: registered new device driver usb [ 0.927142] pps_core: LinuxPPS API ver. 1 registered [ 0.928000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.928050] PTP clock support registered [ 0.931165] EDAC MC: Ver: 3.0.0 [ 0.932112] PCI: Using ACPI for IRQ routing [ 0.933000] NetLabel: Initializing [ 0.934007] NetLabel: domain hash size = 128 [ 0.935007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.937074] NetLabel: unlabeled traffic allowed by default [ 0.939269] vgaarb: loaded [ 0.942837] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.946019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.961095] clocksource: Switched to clocksource kvm-clock [ 1.099748] VFS: Disk quotas dquot_6.6.0 [ 1.101355] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.104185] *** VALIDATE ramfs *** [ 1.105407] *** VALIDATE hugetlbfs *** [ 1.106955] pnp: PnP ACPI init [ 1.109391] pnp: PnP ACPI: found 6 devices [ 1.130491] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.133773] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.135995] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.138462] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.140911] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.143255] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.146043] NET: Registered protocol family 2 [ 1.148487] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.153671] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.157025] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.162161] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.166229] TCP: Hash tables configured (established 65536 bind 65536) [ 1.168966] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.171741] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.174837] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.177499] NET: Registered protocol family 1 [ 1.179920] RPC: Registered named UNIX socket transport module. [ 1.182193] RPC: Registered udp transport module. [ 1.184065] RPC: Registered tcp transport module. [ 1.185709] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.188595] NET: Registered protocol family 44 [ 1.189953] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.192318] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.194206] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.195809] PCI: CLS 0 bytes, default 64 [ 1.196935] Unpacking initramfs... [ 4.036644] debug: unmapping init [mem 0xffff8fec7cc54000-0xffff8fec7ffbffff] [ 4.042822] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.044743] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.048215] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.018617] Initialise system trusted keyrings [ 5.028513] Key type blacklist registered [ 5.030200] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.047816] zbud: loaded [ 5.058240] *** VALIDATE nfs *** [ 5.059686] *** VALIDATE nfs4 *** [ 5.065238] pstore: using deflate compression [ 5.068948] Platform Keyring initialized [ 5.295909] NET: Registered protocol family 38 [ 5.297742] Key type asymmetric registered [ 5.299156] Asymmetric key parser 'x509' registered [ 5.302367] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.313794] io scheduler mq-deadline registered [ 5.315425] io scheduler kyber registered [ 5.317649] io scheduler bfq registered [ 5.319743] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.323931] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.328213] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.331271] ACPI: Power Button [PWRF] [ 5.443489] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.654650] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.087616] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.295941] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.892433] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.961536] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.018971] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.041113] Non-volatile memory driver v1.3 [ 7.042460] Linux agpgart interface v0.103 [ 7.086784] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 7.093608] vda: detected capacity change from 0 to 68571136 [ 7.114456] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.120131] vdb: detected capacity change from 0 to 1073741824 [ 7.146177] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.150935] vdc: detected capacity change from 0 to 2621440000 [ 7.181415] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.183953] vdd: detected capacity change from 0 to 2621440000 [ 7.225253] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.227790] vde: detected capacity change from 0 to 4294967296 [ 7.244232] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.246462] vdf: detected capacity change from 0 to 4294967296 [ 7.256599] libphy: Fixed MDIO Bus: probed [ 7.266967] usbcore: registered new interface driver usbserial_generic [ 7.272419] usbserial: USB Serial support registered for generic [ 7.274740] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.286442] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.287802] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.294130] mousedev: PS/2 mouse device common for all mice [ 7.300778] rtc_cmos 00:05: RTC can wake from S4 [ 7.302618] rtc_cmos 00:05: registered as rtc0 [ 7.305541] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.312311] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.313985] intel_pstate: CPU model not supported [ 7.324160] hid: raw HID events driver (C) Jiri Kosina [ 7.326167] usbcore: registered new interface driver usbhid [ 7.335901] usbhid: USB HID core driver [ 7.338175] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.340719] drop_monitor: Initializing network drop monitor service [ 7.342406] Initializing XFRM netlink socket [ 7.343788] NET: Registered protocol family 10 [ 7.353435] Segment Routing with IPv6 [ 7.354695] NET: Registered protocol family 17 [ 7.356393] mpls_gso: MPLS GSO support [ 7.365389] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.372307] RAS: Correctable Errors collector initialized. [ 7.377285] AVX version of gcm_enc/dec engaged. [ 7.383095] AES CTR mode by8 optimization enabled [ 7.541398] sched_clock: Marking stable (7541382161, 0)->(9424775475, -1883393314) [ 7.544461] registered taskstats version 1 [ 7.547812] Loading compiled-in X.509 certificates [ 7.551993] zswap: loaded using pool lzo/zbud [ 7.617128] Key type big_key registered [ 7.656833] Key type encrypted registered [ 7.665484] ima: No TPM chip found, activating TPM-bypass! [ 7.669826] ima: Allocated hash algorithm: sha1 [ 7.672681] ima: No architecture policies found [ 7.680982] evm: Initialising EVM extended attributes: [ 7.683095] evm: security.selinux [ 7.684308] evm: security.ima [ 7.685581] evm: security.capability [ 7.690162] evm: HMAC attrs: 0x1 [ 7.695284] rtc_cmos 00:05: setting system clock to 2025-12-03 06:49:53 UTC (1764744593) [ 7.713400] debug: unmapping init [mem 0xffffffffa9203000-0xffffffffa93fffff] [ 7.722622] debug: unmapping init [mem 0xffffffffa7f82000-0xffffffffa8258fff] [ 7.725325] Write protecting the kernel read-only data: 28672k [ 7.733111] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 7.736021] debug: unmapping init [mem 0xffffffffa6f14000-0xffffffffa6ffffff] [ 7.828835] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.851318] systemd[1]: Detected virtualization kvm. [ 7.852903] systemd[1]: Detected architecture x86-64. [ 7.857992] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.902514] systemd[1]: No hostname configured. [ 7.903771] systemd[1]: Set hostname to . [ 7.910753] random: systemd: uninitialized urandom read (16 bytes read) [ 7.912881] systemd[1]: Initializing machine ID from random generator. [ 7.976116] random: ln: uninitialized urandom read (6 bytes read) [ 8.225721] random: systemd: uninitialized urandom read (16 bytes read) [ 8.227764] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.235560] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 8.243261] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.509162] device-mapper: uevent: version 1.0.3 [ 9.511504] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.680492] virtio_net virtio0 ens2: renamed from eth0 [ 10.733852] random: fast init done [ 12.519469] scsi host0: ata_piix [ 12.591983] scsi host1: ata_piix [ 12.593231] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.609913] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.209726] random: crng init done [ 16.210942] random: 7 urandom warning(s) missed due to ratelimiting [ 19.083163] dracut-initqueue[602]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.681719] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.269433] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.777580] SELinux: Disabled at runtime. [ 23.861455] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.869412] systemd[1]: Detected virtualization kvm. [ 23.871651] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.176025] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.183606] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.200467] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.204167] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.211561] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.226124] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.259491] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount[ 25.358096] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 26.333764] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.968742] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.081285] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.160807] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.196713] EDAC sbridge: Ver: 1.1.2 [ 30.767395] Key type dns_resolver registered [ 31.266205] NFS: Registering the id_resolver key type [ 31.269716] Key type id_resolver registered [ 31.271464] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg308-server login: [ 54.661122] libcfs: loading out-of-tree module taints kernel. [ 54.676675] Key type ._llcrypt registered [ 54.678177] Key type .llcrypt registered [ 54.715973] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 61.794223] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 62.408177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.415431] alg: No test for adler32 (adler32-zlib) [ 63.389603] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 63.657839] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 65.263144] Key type lgssc registered [ 65.856471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.778307] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.179930] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.075087] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.693930] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.430626] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.864632] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 91.328144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.354697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 91.362654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 92.455150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 92.470095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 92.509832] Lustre: lustre-MDT0000: new disk, initializing [ 92.535930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.543760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 93.887455] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.132544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.165598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.202701] Lustre: 6510:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 99.221407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 99.223830] Lustre: Skipped 1 previous similar message [ 99.266656] Lustre: lustre-MDT0001: new disk, initializing [ 99.294610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.308227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 99.311992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 100.717314] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.981647] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 106.159386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.179826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.265087] Lustre: lustre-OST0000: new disk, initializing [ 106.266944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 106.283955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 106.861842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 106.863807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 106.890157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 108.179547] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.531711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.559603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.594434] Lustre: lustre-OST0001: new disk, initializing [ 113.595852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 113.612273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 115.634686] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.907064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 116.911274] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 116.923502] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 121.369870] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 125.991762] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 132.820739] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing check_logdir /tmp/testlogs/ [ 134.184600] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing yml_node [ 135.582748] Lustre: DEBUG MARKER: Client: 2.17.0.RC1 [ 136.420249] Lustre: DEBUG MARKER: MDS: 2.17.0.RC1 [ 137.263321] Lustre: DEBUG MARKER: OSS: 2.17.0.RC1 [ 137.779323] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Dec 3 01:52:02 EST 2025 [ 143.501442] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 143.974788] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 155.615612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 155.618923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.622975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.176071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 158.176234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.179229] Lustre: Skipped 2 previous similar messages [ 158.180656] Lustre: Skipped 2 previous similar messages [ 161.659643] LustreError: 12210:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 161.680927] Lustre: server umount lustre-MDT0000 complete [ 162.807855] LustreError: 6500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764744748 with bad export cookie 2219636210128505195 [ 162.808884] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 162.811778] LustreError: 6500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 163.295595] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.295873] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 163.298508] Lustre: Skipped 1 previous similar message [ 163.301583] Lustre: Skipped 2 previous similar messages [ 167.327494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 167.329170] Lustre: Skipped 1 previous similar message [ 172.447774] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 172.447826] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 172.451679] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 172.453164] Lustre: Skipped 1 previous similar message [ 177.119192] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 177.163659] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 177.165421] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 177.218786] Lustre: server umount lustre-MDT0001 complete [ 178.534247] LustreError: 12615:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 178.535944] LustreError: 12615:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 178.555628] Lustre: server umount lustre-OST0000 complete [ 179.912472] Lustre: server umount lustre-OST0001 complete [ 181.979387] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 184.967743] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 188.435681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.557248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.520048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.479397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.818897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.838852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.922156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.933984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.969160] Lustre: lustre-MDT0000: new disk, initializing [ 196.992715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.999362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 198.263456] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 202.326551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.351502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 202.384898] Lustre: 14993:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.480618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.747794] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 206.784820] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 207.955431] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 210.375919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 210.397882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 210.498967] Lustre: lustre-OST0000: new disk, initializing [ 210.500515] Lustre: Skipped 1 previous similar message [ 210.502676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.505087] Lustre: Skipped 2 previous similar messages [ 210.536177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.537674] Lustre: Skipped 1 previous similar message [ 212.350560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.354747] Lustre: Skipped 1 previous similar message [ 212.357742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.403781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.502588] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 215.536944] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 216.827445] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 216.900217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 218.208793] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 218.274353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 222.689612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.690719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 222.693246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 222.694133] LustreError: Skipped 1 previous similar message [ 222.695446] Lustre: Skipped 1 previous similar message [ 224.940202] LustreError: 17930:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 224.943181] LustreError: 17930:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 224.981827] Lustre: server umount lustre-OST0000 complete [ 227.808963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 227.812771] Lustre: Skipped 2 previous similar messages [ 232.260735] LustreError: 18131:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 232.262938] LustreError: 18131:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 232.292148] Lustre: server umount lustre-MDT0000 complete [ 232.929195] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.936027] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 233.628210] LustreError: 15784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764744819 with bad export cookie 2219636210128506714 [ 233.630703] LustreError: 15784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 233.631578] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 233.746667] Lustre: server umount lustre-MDT0001 complete [ 234.886860] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:53:39 (1764744819) [ 236.989322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.132592] LustreError: 18858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 237.154500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.363942] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 241.199326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 242.601888] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 244.392298] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 245.589069] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 247.909076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 249.885184] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 251.648091] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 266.721386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 266.723657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.725755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 266.728550] Lustre: Skipped 1 previous similar message [ 266.731491] Lustre: Skipped 3 previous similar messages [ 270.635695] LustreError: 21125:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 270.637530] LustreError: 21125:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 270.670690] Lustre: server umount lustre-OST0000 complete [ 272.351757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 286.175116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 286.235287] Lustre: server umount lustre-MDT0000 complete [ 287.200097] LustreError: 18858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.204346] LustreError: 18858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 287.429490] LustreError: 18836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764744873 with bad export cookie 2219636210128507596 [ 287.431968] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 287.432464] LustreError: 18836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 287.500579] LustreError: 21530:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 287.502760] LustreError: 21530:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 287.548924] Lustre: server umount lustre-MDT0001 complete [ 290.967041] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 291.962173] Key type lgssc unregistered [ 292.075324] LNet: 22060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.077426] LNetError: 22060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 292.085342] LNet: Removed LNI 192.168.203.108@tcp [ 292.394117] Key type .llcrypt unregistered [ 292.395704] Key type ._llcrypt unregistered [ 300.881078] Key type ._llcrypt registered [ 300.882258] Key type .llcrypt registered [ 300.915413] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 301.224296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 301.270086] alg: No test for adler32 (adler32-zlib) [ 302.113214] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 302.186435] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 303.767134] Key type lgssc registered [ 304.085267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 306.483235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 306.488832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 307.631130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 308.873700] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 311.728300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 311.850216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 313.171721] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 314.956205] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 316.153639] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 318.420801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 318.541767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 320.552150] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 322.335698] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 332.898920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 337.377263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 337.380153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.385397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 338.401307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 338.401381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 338.401442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.410937] Lustre: Skipped 1 previous similar message [ 339.119213] LustreError: 26139:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 339.162303] Lustre: server umount lustre-OST0000 complete [ 342.495964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 342.499116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.503684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.588538] LustreError: 26341:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 342.591208] LustreError: 26341:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 342.619689] Lustre: server umount lustre-MDT0000 complete [ 343.520063] LustreError: 23429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.526593] LustreError: 23429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 343.968480] LustreError: 23410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764744929 with bad export cookie 3824460631894095588 [ 343.971370] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.976964] LustreError: 23410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 344.039333] LustreError: 26541:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 344.041083] LustreError: 26541:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 344.094701] Lustre: server umount lustre-MDT0001 complete [ 347.787694] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 348.846916] Key type lgssc unregistered [ 348.975458] LNet: 27072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.978985] LNetError: 27072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.990325] LNet: Removed LNI 192.168.203.108@tcp [ 349.296173] Key type .llcrypt unregistered [ 349.297491] Key type ._llcrypt unregistered [ 357.839043] Key type ._llcrypt registered [ 357.839900] Key type .llcrypt registered [ 357.871909] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 358.203393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 358.228433] alg: No test for adler32 (adler32-zlib) [ 359.073111] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 359.154073] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 360.735112] Key type lgssc registered [ 361.085442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 363.595711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 363.603045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.762918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 366.127235] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 369.098077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 369.222918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 370.550482] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 372.421796] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 373.675118] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 378.920124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 379.076531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 381.212628] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 383.120521] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 390.309225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 398.817324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.817878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 398.823585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 398.827296] Lustre: Skipped 1 previous similar message [ 402.029420] LustreError: 31064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 402.064103] Lustre: server umount lustre-OST0000 complete [ 403.936531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.937076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 403.941711] Lustre: Skipped 2 previous similar messages [ 409.056411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 409.058813] Lustre: Skipped 2 previous similar messages [ 409.467800] LustreError: 31265:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 409.469792] LustreError: 31265:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 409.501679] Lustre: server umount lustre-MDT0000 complete [ 410.866588] LustreError: 30413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764744996 with bad export cookie 11609079780248495690 [ 410.869892] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.871216] LustreError: 30413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 410.944596] LustreError: 31467:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 410.946735] LustreError: 31467:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 410.999407] Lustre: server umount lustre-MDT0001 complete [ 414.551088] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 415.691871] Key type lgssc unregistered [ 415.837495] LNet: 31998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.840663] LNetError: 31998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 415.847848] LNet: Removed LNI 192.168.203.108@tcp [ 416.188667] Key type .llcrypt unregistered [ 416.190454] Key type ._llcrypt unregistered [ 425.507030] Key type ._llcrypt registered [ 425.508082] Key type .llcrypt registered [ 425.554751] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 425.876593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 425.885435] alg: No test for adler32 (adler32-zlib) [ 426.749619] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 426.836066] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 428.423191] Key type lgssc registered [ 428.823896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 431.666203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 431.675364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.846183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.282457] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 437.241216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 437.374804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.833077] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 440.713029] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 441.999640] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 444.299037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 444.411736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.375830] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 448.300665] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 457.763965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 462.817806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.818844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.820305] LustreError: Skipped 1 previous similar message [ 462.827171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.829233] Lustre: Skipped 1 previous similar message [ 464.430109] LustreError: 35634:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 464.474866] Lustre: server umount lustre-OST0000 complete [ 467.901637] LustreError: 35834:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 467.903944] LustreError: 35834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 467.929213] Lustre: server umount lustre-MDT0000 complete [ 467.937426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.937947] LustreError: 33365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.941483] Lustre: Skipped 2 previous similar messages [ 467.946225] LustreError: 33365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 469.266199] LustreError: 34985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745055 with bad export cookie 13492462808982681519 [ 469.269493] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.269954] LustreError: 34985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 469.342859] LustreError: 36037:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 469.345228] LustreError: 36037:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 469.397606] Lustre: server umount lustre-MDT0001 complete [ 473.029788] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 474.070964] Key type lgssc unregistered [ 474.211475] LNet: 36568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.214747] LNetError: 36568:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.227319] LNet: Removed LNI 192.168.203.108@tcp [ 474.513514] Key type .llcrypt unregistered [ 474.514474] Key type ._llcrypt unregistered [ 483.091615] Key type ._llcrypt registered [ 483.092673] Key type .llcrypt registered [ 483.127979] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 483.492571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 483.513691] alg: No test for adler32 (adler32-zlib) [ 484.359049] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 484.441364] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 486.015146] Key type lgssc registered [ 486.385437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 488.987970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 488.995322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.146226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.436445] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 494.269488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 494.380225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 495.666690] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 497.446680] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 498.585642] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 500.755606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 500.861238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.701409] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 504.511747] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 514.214487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 519.649217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.650450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.656703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.659062] Lustre: Skipped 1 previous similar message [ 523.246083] LustreError: 40203:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 523.282799] Lustre: server umount lustre-OST0000 complete [ 536.121782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.127253] Lustre: Skipped 1 previous similar message [ 536.129870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.132235] Lustre: Skipped 2 previous similar messages [ 539.232214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.234211] Lustre: Skipped 1 previous similar message [ 542.330542] LustreError: 40578:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 542.332572] LustreError: 40578:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 542.357062] Lustre: server umount lustre-MDT0000 complete [ 543.617363] LustreError: 37918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745129 with bad export cookie 11405563189691187302 [ 543.620148] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.621553] LustreError: 37918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 549.755445] LustreError: 40779:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 549.757597] LustreError: 40779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 549.805714] Lustre: server umount lustre-MDT0001 complete [ 553.309134] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 554.260038] Key type lgssc unregistered [ 554.396648] LNet: 41311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.398758] LNetError: 41311:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 554.410371] LNet: Removed LNI 192.168.203.108@tcp [ 554.679231] Key type .llcrypt unregistered [ 554.680372] Key type ._llcrypt unregistered [ 563.185633] Key type ._llcrypt registered [ 563.186909] Key type .llcrypt registered [ 563.222405] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 563.522264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 563.533419] alg: No test for adler32 (adler32-zlib) [ 564.392517] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 564.473573] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 566.055195] Key type lgssc registered [ 566.428905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 569.137785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 569.144710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.312951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.595365] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 574.453259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 574.581955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 575.942471] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 577.857772] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 579.130338] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 581.591204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 581.715067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.756771] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 585.702138] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 596.070193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 600.544334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 600.547731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.551235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.501336] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.8@tcp (stopping) [ 601.570001] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.863302] LustreError: 44947:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 604.895206] Lustre: server umount lustre-MDT0000 complete [ 606.181383] LustreError: 42705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745191 with bad export cookie 62257089868235761 [ 606.184566] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.185479] LustreError: 42705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 606.614269] LustreError: 42675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.621356] LustreError: 42675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 606.623359] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.8@tcp (stopping) [ 606.627343] Lustre: Skipped 3 previous similar messages [ 606.688500] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.693166] Lustre: Skipped 1 previous similar message [ 611.733449] LustreError: 42676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.733505] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.8@tcp (stopping) [ 611.738907] LustreError: 42676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 611.741740] Lustre: Skipped 1 previous similar message [ 612.285739] LustreError: 45150:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 612.287832] LustreError: 45150:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 612.335270] Lustre: server umount lustre-MDT0001 complete [ 630.318320] LustreError: 45374:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 630.321182] LustreError: 45374:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 630.365346] Lustre: server umount lustre-OST0000 complete [ 635.809821] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 636.836972] Key type lgssc unregistered [ 636.977467] LNet: 46211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.979913] LNetError: 46211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.987428] LNet: Removed LNI 192.168.203.108@tcp [ 637.313526] Key type .llcrypt unregistered [ 637.314657] Key type ._llcrypt unregistered [ 646.824995] Key type ._llcrypt registered [ 646.825855] Key type .llcrypt registered [ 646.857803] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 647.255039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.261241] alg: No test for adler32 (adler32-zlib) [ 648.116168] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 648.199870] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 649.775173] Key type lgssc registered [ 650.110844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.586673] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 652.592333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 667.999306] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 673.119202] LustreError: 47556:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 699.743213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.471355] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 703.188976] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 867.063835] LustreError: 48275:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 867.102906] Lustre: server umount lustre-OST0000 complete [ 872.243051] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 873.204646] Key type lgssc unregistered [ 873.319326] LNet: 49111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.321657] LNetError: 49111:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.333376] LNet: Removed LNI 192.168.203.108@tcp [ 873.600799] Key type .llcrypt unregistered [ 873.602225] Key type ._llcrypt unregistered [ 881.963198] Key type ._llcrypt registered [ 881.964331] Key type .llcrypt registered [ 882.000781] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 882.319411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.355045] alg: No test for adler32 (adler32-zlib) [ 883.203363] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 883.286224] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 884.863150] Key type lgssc registered [ 885.196717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.601175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.607087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.753429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.954331] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 892.631292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 892.757342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.979387] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 895.656739] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.809836] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 898.946908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 899.051238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.398712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 900.773262] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 902.474596] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 905.695691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 905.697518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.701081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 907.745691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 907.746367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 907.749590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.227688] LustreError: 52724:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 909.263815] Lustre: server umount lustre-OST0000 complete [ 912.864486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.865250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 912.867951] Lustre: Skipped 1 previous similar message [ 912.869723] Lustre: Skipped 1 previous similar message [ 916.537851] LustreError: 52926:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 916.539438] LustreError: 52926:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 916.564791] Lustre: server umount lustre-MDT0000 complete [ 917.712177] LustreError: 50462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745503 with bad export cookie 14207109300986117834 [ 917.714097] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.716637] LustreError: 50462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 917.785645] LustreError: 53128:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 917.788202] LustreError: 53128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 917.834786] Lustre: server umount lustre-MDT0001 complete [ 921.187179] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 922.170688] Key type lgssc unregistered [ 922.299366] LNet: 53659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.302972] LNetError: 53659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.312313] LNet: Removed LNI 192.168.203.108@tcp [ 922.604380] Key type .llcrypt unregistered [ 922.606071] Key type ._llcrypt unregistered [ 930.771518] Key type ._llcrypt registered [ 930.772849] Key type .llcrypt registered [ 930.808339] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 931.203797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 931.210312] alg: No test for adler32 (adler32-zlib) [ 932.051517] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 932.126936] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 933.703106] Key type lgssc registered [ 934.031650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.447682] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 936.454454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 951.903305] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 957.023189] LustreError: 55001:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 983.647367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.808909] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 987.821125] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 990.075985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.247474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.272305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 991.576452] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 994.527497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 994.692721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.082487] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 997.961324] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 999.173723] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 999.904666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.907347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.910852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1000.415827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1000.417724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.421064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.000973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.002491] Lustre: Skipped 1 previous similar message [ 1009.121189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.124302] Lustre: Skipped 1 previous similar message [ 1014.239130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1014.243631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1014.246048] Lustre: Skipped 1 previous similar message [ 1014.257168] LustreError: 57280:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1014.289203] Lustre: server umount lustre-OST0000 complete [ 1028.543707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.548619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.550318] Lustre: Skipped 1 previous similar message [ 1034.748337] LustreError: 57660:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1034.750671] LustreError: 57660:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1034.775970] Lustre: server umount lustre-MDT0000 complete [ 1036.004053] LustreError: 55009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745621 with bad export cookie 6956161629450157812 [ 1036.007344] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1036.008705] LustreError: 55009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1042.107213] LustreError: 57862:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1042.108895] LustreError: 57862:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1042.158724] Lustre: server umount lustre-MDT0001 complete [ 1045.649397] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1046.617951] Key type lgssc unregistered [ 1046.738342] LNet: 58393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.740290] LNetError: 58393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.749304] LNet: Removed LNI 192.168.203.108@tcp [ 1047.022485] Key type .llcrypt unregistered [ 1047.023429] Key type ._llcrypt unregistered [ 1055.729363] Key type ._llcrypt registered [ 1055.730277] Key type .llcrypt registered [ 1055.768263] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1056.141389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.160146] alg: No test for adler32 (adler32-zlib) [ 1057.017913] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1057.113916] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1058.703180] Key type lgssc registered [ 1059.083417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.658789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.664126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.817569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.143197] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1067.099354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1067.228869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.569662] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1070.379384] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1071.621743] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1073.887267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1074.002215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.867381] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1077.784744] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1087.334364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1092.578863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.578909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.584742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1092.588709] Lustre: Skipped 1 previous similar message [ 1096.300148] LustreError: 62030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1096.336418] Lustre: server umount lustre-OST0000 complete [ 1097.696217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.696757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.701016] Lustre: Skipped 2 previous similar messages [ 1097.703601] Lustre: Skipped 1 previous similar message [ 1102.816256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.818536] Lustre: Skipped 1 previous similar message [ 1103.744726] LustreError: 62233:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1103.748071] LustreError: 62233:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1103.779511] Lustre: server umount lustre-MDT0000 complete [ 1105.084143] LustreError: 59745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745690 with bad export cookie 14353381988421540373 [ 1105.087131] LustreError: 59745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1105.087225] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.155317] LustreError: 62435:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1105.158044] LustreError: 62435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1105.208180] Lustre: server umount lustre-MDT0001 complete [ 1108.918411] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1110.004277] Key type lgssc unregistered [ 1110.150444] LNet: 62965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.153882] LNetError: 62965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.160476] LNet: Removed LNI 192.168.203.108@tcp [ 1110.456361] Key type .llcrypt unregistered [ 1110.457329] Key type ._llcrypt unregistered [ 1122.513530] Key type ._llcrypt registered [ 1122.514453] Key type .llcrypt registered [ 1122.551926] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1123.002117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.008102] alg: No test for adler32 (adler32-zlib) [ 1123.866708] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1123.948908] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1125.527173] Key type lgssc registered [ 1125.889850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1128.458074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1128.464497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.618049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.885465] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1133.679796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1133.808547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1135.108894] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1136.922373] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1138.105320] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1140.410085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1140.519524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.376067] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1144.165610] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1154.850808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1157.108675] Lustre: Failing over lustre-MDT0000 [ 1157.283235] LustreError: 66895:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1157.311100] Lustre: server umount lustre-MDT0000 complete [ 1159.648093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1159.651095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.656520] LustreError: 64625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.664364] LustreError: 64625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1160.160409] LustreError: 64625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.162123] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.166218] LustreError: 64625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1160.461899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.499214] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.550721] Lustre: *** cfs_fail_loc=135, val=0*** [ 1160.591290] LustreError: 67403:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1160.593170] LustreError: 67403:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.108@tcp: cfg command failed: rc = -2 [ 1160.595789] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1160.595789] [ 1160.598688] LustreError: MGC192.168.203.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1160.602023] LustreError: 67391:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1160.603926] LustreError: 67391:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1160.606293] LustreError: 67391:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1160.615512] Lustre: server umount lustre-MDT0000 complete [ 1160.616885] LustreError: 67391:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1162.875465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.915815] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.022075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.043164] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1164.278475] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1165.205677] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1168.362700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1168.370707] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1168.387943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1169.375820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1169.378716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.384261] Lustre: Skipped 1 previous similar message [ 1169.386438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.476751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1173.476767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.478571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.470117] LustreError: 68210:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1175.473440] LustreError: 68210:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1175.508993] Lustre: server umount lustre-OST0000 complete [ 1178.593532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1178.593899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.596804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.600200] Lustre: Skipped 1 previous similar message [ 1178.603661] Lustre: Skipped 2 previous similar messages [ 1183.712698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.715389] Lustre: Skipped 1 previous similar message [ 1188.832699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1188.835198] Lustre: Skipped 1 previous similar message [ 1191.391151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1191.427056] LustreError: 68411:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1191.428778] LustreError: 68411:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1191.452717] Lustre: server umount lustre-MDT0000 complete [ 1192.674448] LustreError: 64608:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745778 with bad export cookie 13083208952316492962 [ 1192.677725] LustreError: 64608:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1192.677897] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.797107] Lustre: server umount lustre-MDT0001 complete [ 1196.538544] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1197.672237] Key type lgssc unregistered [ 1197.809403] LNet: 69146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.811892] LNetError: 69146:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.820422] LNet: Removed LNI 192.168.203.108@tcp [ 1198.154718] Key type .llcrypt unregistered [ 1198.156553] Key type ._llcrypt unregistered [ 1206.952944] Key type ._llcrypt registered [ 1206.953940] Key type .llcrypt registered [ 1206.987107] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1207.434185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1207.447092] alg: No test for adler32 (adler32-zlib) [ 1208.291381] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1208.371536] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1209.951195] Key type lgssc registered [ 1210.317670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1212.946443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1212.952201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.120791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.543447] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1218.560855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1218.702444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1220.083495] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1221.913624] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1223.155886] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1225.435528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1225.549377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.376926] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1229.228318] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1238.886896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1242.054712] Lustre: Failing over lustre-MDT0000 [ 1242.171209] LustreError: 72792:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1242.197390] Lustre: server umount lustre-MDT0000 complete [ 1244.128811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.131561] LustreError: 70510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.131925] Lustre: Skipped 2 previous similar messages [ 1244.135492] LustreError: 70510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1245.084599] LustreError: 70512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.089226] LustreError: 70512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1245.420182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.463812] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.533354] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1245.607504] LustreError: 73300:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1245.610857] LustreError: 73300:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.108@tcp: cfg command failed: rc = -9 [ 1245.615065] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1245.615065] [ 1245.619109] LustreError: MGC192.168.203.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1245.624746] LustreError: 73288:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1245.627866] LustreError: 73288:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1245.631280] LustreError: 73288:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1245.642352] Lustre: server umount lustre-MDT0000 complete [ 1245.643530] LustreError: 73288:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1247.864661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.899684] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.979290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.995049] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1249.309665] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1250.197527] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1253.352569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1253.358238] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1253.380322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1254.367845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1254.370292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.374741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.467676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1258.469301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1258.470724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.461086] LustreError: 74109:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1260.462739] LustreError: 74109:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1260.498271] Lustre: server umount lustre-OST0000 complete [ 1263.585444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.585601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1263.585991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.585995] Lustre: Skipped 1 previous similar message [ 1263.588543] Lustre: Skipped 1 previous similar message [ 1268.704090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.706600] Lustre: Skipped 2 previous similar messages [ 1273.824218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.826087] Lustre: Skipped 1 previous similar message [ 1275.871150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1275.906450] LustreError: 74310:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1275.908083] LustreError: 74310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1275.932727] Lustre: server umount lustre-MDT0000 complete [ 1277.256196] LustreError: 70495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745863 with bad export cookie 201980653414900781 [ 1277.259532] LustreError: 70495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1277.259558] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.380636] Lustre: server umount lustre-MDT0001 complete [ 1281.533715] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1282.572831] Key type lgssc unregistered [ 1282.701459] LNet: 75044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.703906] LNetError: 75044:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.710466] LNet: Removed LNI 192.168.203.108@tcp [ 1283.041073] Key type .llcrypt unregistered [ 1283.043049] Key type ._llcrypt unregistered [ 1287.273196] Key type ._llcrypt registered [ 1287.274271] Key type .llcrypt registered [ 1287.314733] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1287.417408] Key type .llcrypt unregistered [ 1287.418378] Key type ._llcrypt unregistered [ 1297.075896] Key type ._llcrypt registered [ 1297.078947] Key type .llcrypt registered [ 1297.142630] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1297.534512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1297.584612] alg: No test for adler32 (adler32-zlib) [ 1298.512866] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1298.654042] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1300.263183] Key type lgssc registered [ 1300.875125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1305.622348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1305.636043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.944490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1309.480435] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1314.477944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1314.717852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1317.650639] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1321.321215] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1324.959475] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1331.626217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1332.046810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.127313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 1338.394348] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1343.497294] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1353.697722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.701090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.708731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1354.724275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1354.725502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1354.732124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.962827] LustreError: 79132:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1356.028398] Lustre: server umount lustre-OST0000 complete [ 1359.842190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.846561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.853133] Lustre: Skipped 1 previous similar message [ 1359.859228] Lustre: Skipped 2 previous similar messages [ 1364.102904] LustreError: 79334:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1364.106863] LustreError: 79334:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1364.158499] Lustre: server umount lustre-MDT0000 complete [ 1364.960886] LustreError: 76881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.968200] LustreError: 76881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1366.097757] LustreError: 77607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745951 with bad export cookie 13942487862272518898 [ 1366.100394] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.102734] LustreError: 77607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1366.204834] LustreError: 79535:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1366.208621] LustreError: 79535:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1366.285444] Lustre: server umount lustre-MDT0001 complete [ 1370.408456] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1371.645159] Key type lgssc unregistered [ 1371.791064] LNet: 80067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.794512] LNetError: 80067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.807438] LNet: Removed LNI 192.168.203.108@tcp [ 1372.158977] Key type .llcrypt unregistered [ 1372.160822] Key type ._llcrypt unregistered [ 1382.796814] Key type ._llcrypt registered [ 1382.797880] Key type .llcrypt registered [ 1382.849493] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1383.340896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1383.358180] alg: No test for adler32 (adler32-zlib) [ 1384.323147] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1384.467930] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1386.095176] Key type lgssc registered [ 1386.580122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1389.821753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1389.833520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.079250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.898151] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1396.811264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1397.025861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1398.791424] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1401.166738] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1402.782380] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1406.018376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1406.194229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.729167] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1411.279904] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1415.407929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 1420.768680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.770828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.775244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.782397] Lustre: Skipped 1 previous similar message [ 1420.787247] Lustre: Skipped 1 previous similar message [ 1421.108178] LustreError: 83680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1421.181240] Lustre: server umount lustre-OST0000 complete [ 1422.816401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1422.819858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.825920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.832745] LustreError: 83882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1424.834769] LustreError: 83882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1424.876277] Lustre: server umount lustre-MDT0000 complete [ 1425.888980] LustreError: 81438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.889240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.895843] LustreError: 81438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1426.397266] LustreError: 81418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746012 with bad export cookie 354701810997649308 [ 1426.399796] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1426.402544] LustreError: 81418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1426.473816] LustreError: 84084:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1426.475724] LustreError: 84084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1426.557128] Lustre: server umount lustre-MDT0001 complete [ 1430.929744] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1432.050384] Key type lgssc unregistered [ 1432.188515] LNet: 84615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.192231] LNetError: 84615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.200408] LNet: Removed LNI 192.168.203.108@tcp [ 1432.506566] Key type .llcrypt unregistered [ 1432.507977] Key type ._llcrypt unregistered [ 1442.229987] Key type ._llcrypt registered [ 1442.231365] Key type .llcrypt registered [ 1442.281528] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1442.795038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1442.804980] alg: No test for adler32 (adler32-zlib) [ 1443.675905] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1443.770668] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1445.359225] Key type lgssc registered [ 1445.830405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1448.835224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1448.842822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.026739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.645263] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1455.846495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1456.036733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.994590] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1460.810448] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1462.660674] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1466.039591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1466.228791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.365527] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1472.401521] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1474.411314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 1489.888904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1489.892210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.898519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1490.912994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1490.916857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1490.922568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.427126] LustreError: 88253:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1493.481642] Lustre: server umount lustre-OST0000 complete [ 1496.034277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.043395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.046764] Lustre: Skipped 2 previous similar messages [ 1501.155647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.160909] Lustre: Skipped 1 previous similar message [ 1501.588279] LustreError: 88455:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1501.590310] LustreError: 88455:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1501.623458] Lustre: server umount lustre-MDT0000 complete [ 1503.383569] LustreError: 87602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746089 with bad export cookie 15192819014107290826 [ 1503.385925] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1503.389100] LustreError: 87602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1503.471139] LustreError: 88657:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1503.473710] LustreError: 88657:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1503.536939] Lustre: server umount lustre-MDT0001 complete [ 1507.617595] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1508.749524] Key type lgssc unregistered [ 1508.893581] LNet: 89188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.897166] LNetError: 89188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.905780] LNet: Removed LNI 192.168.203.108@tcp [ 1509.235935] Key type .llcrypt unregistered [ 1509.237182] Key type ._llcrypt unregistered [ 1518.774506] Key type ._llcrypt registered [ 1518.775799] Key type .llcrypt registered [ 1518.817211] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1519.129935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1519.164046] alg: No test for adler32 (adler32-zlib) [ 1520.008505] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1520.088777] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1521.671144] Key type lgssc registered [ 1522.044868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.659534] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1524.664729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1540.127299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1545.247282] LustreError: 90529:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 1571.871173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.690629] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1575.565283] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1577.626784] LustreError: 91385:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1577.652251] Lustre: server umount lustre-OST0000 complete [ 1580.465409] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:16:05 (1764746165) [ 1582.821177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.974827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.378615] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1587.092637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1587.197848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1588.349759] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1590.006703] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1591.175724] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1593.271804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1593.313210] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.318721] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1593.372410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.097835] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1596.458682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 1596.753585] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1601.506370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1601.506601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.510767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1601.513297] Lustre: Skipped 1 previous similar message [ 1602.926062] LustreError: 94457:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1602.927774] LustreError: 94457:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1602.960163] Lustre: server umount lustre-OST0000 complete [ 1606.201823] LustreError: 94658:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1606.204022] LustreError: 94658:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1606.229597] Lustre: server umount lustre-MDT0000 complete [ 1606.624277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.630045] LustreError: 92053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.634464] LustreError: 92053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1607.365489] LustreError: 92036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746193 with bad export cookie 14197574380354063412 [ 1607.369449] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1607.492329] Lustre: server umount lustre-MDT0001 complete [ 1611.163176] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1612.235917] Key type lgssc unregistered [ 1612.368337] LNet: 95391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.371488] LNetError: 95391:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.380318] LNet: Removed LNI 192.168.203.108@tcp [ 1612.689245] Key type .llcrypt unregistered [ 1612.690243] Key type ._llcrypt unregistered [ 1621.622224] Key type ._llcrypt registered [ 1621.623212] Key type .llcrypt registered [ 1621.664563] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1622.228369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1622.239642] alg: No test for adler32 (adler32-zlib) [ 1623.096914] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1623.181691] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1624.767195] Key type lgssc registered [ 1625.099823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1627.560502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1627.565942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.716889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.937835] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1632.765474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1632.890729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1634.170628] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1635.964692] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1637.139802] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1639.298586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1639.409078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1641.226427] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1642.477216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 1643.020331] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1647.586164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1647.587094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.592877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.594336] Lustre: Skipped 1 previous similar message [ 1648.621462] LustreError: 99007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1648.651948] Lustre: server umount lustre-OST0000 complete [ 1652.705373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.706940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.708803] Lustre: Skipped 2 previous similar messages [ 1652.711463] Lustre: Skipped 1 previous similar message [ 1656.003732] LustreError: 99209:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1656.005517] LustreError: 99209:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1656.045208] Lustre: server umount lustre-MDT0000 complete [ 1657.313379] LustreError: 96745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746243 with bad export cookie 5844396177597932849 [ 1657.317261] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.317938] LustreError: 96745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1657.381706] LustreError: 99410:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1657.384332] LustreError: 99410:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1657.434301] Lustre: server umount lustre-MDT0001 complete [ 1661.112263] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1662.192583] Key type lgssc unregistered [ 1662.332466] LNet: 99940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.336274] LNetError: 99940:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.342331] LNet: Removed LNI 192.168.203.108@tcp [ 1662.650083] Key type .llcrypt unregistered [ 1662.652085] Key type ._llcrypt unregistered [ 1669.863670] Key type ._llcrypt registered [ 1669.864681] Key type .llcrypt registered [ 1669.902347] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1670.246254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1670.278785] alg: No test for adler32 (adler32-zlib) [ 1671.126700] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1671.206348] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1672.783184] Key type lgssc registered [ 1673.138648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1675.664263] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1675.669691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1691.103315] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1696.223242] LustreError: 101140:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 1722.847247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.838907] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1726.615410] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1728.867918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.962196] LustreError: MGC192.168.203.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1728.965743] LustreError: 102051:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1728.968044] LustreError: 102051:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1728.970669] LustreError: 102051:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1728.998531] Lustre: server umount lustre-MDT0000 complete [ 1728.999819] LustreError: 102051:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1741.791575] LustreError: 102482:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1741.827457] Lustre: server umount lustre-OST0000 complete [ 1744.676899] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 1747.937372] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1751.344765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1753.492975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1755.519969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1757.639338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1759.918165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1759.934798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.012501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1760.026496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1760.062660] Lustre: lustre-MDT0000: new disk, initializing [ 1760.089264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.098170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1761.368795] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1765.410747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1765.432533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1765.467314] Lustre: 104815:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1765.493641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1765.498071] Lustre: Skipped 1 previous similar message [ 1765.541609] Lustre: lustre-MDT0001: new disk, initializing [ 1765.574179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1765.584925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1765.588622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1766.827528] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1769.908044] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1771.141605] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1773.402062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1773.429333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1773.522541] Lustre: lustre-OST0000: new disk, initializing [ 1773.524121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1773.549925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.327143] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1776.053575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1776.056229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1776.085473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1778.379987] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1779.765195] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1779.853591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1781.209368] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1781.302807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1784.047106] LustreError: 107754:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1784.048887] LustreError: 107754:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1784.084478] Lustre: server umount lustre-OST0000 complete [ 1786.337474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1786.338742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.339277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.346847] Lustre: Skipped 1 previous similar message [ 1791.457184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.460751] Lustre: Skipped 2 previous similar messages [ 1795.555249] LustreError: 107954:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1795.557110] LustreError: 107954:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1795.582225] Lustre: server umount lustre-MDT0000 complete [ 1796.801358] LustreError: 106559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746382 with bad export cookie 2279172369168895956 [ 1796.804582] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.804761] LustreError: 106559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1806.308085] LustreError: 108156:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1806.310555] LustreError: 108156:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1806.360805] Lustre: server umount lustre-MDT0001 complete [ 1808.999152] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:19:54 (1764746394) [ 1810.024470] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1815.252212] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 1818.426764] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1821.984582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1824.049798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1825.988169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1827.982564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1830.148978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1830.173410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.256854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1830.266630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1830.300654] Lustre: lustre-MDT0000: new disk, initializing [ 1830.324983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.332730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1831.689109] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1835.765183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1835.795601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1835.831215] Lustre: 111286:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1835.860168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1835.862644] Lustre: Skipped 1 previous similar message [ 1835.900864] Lustre: lustre-MDT0001: new disk, initializing [ 1835.947217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1835.950045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1837.270792] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1840.260510] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1841.494606] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1843.828045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1843.854173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1843.965491] Lustre: lustre-OST0000: new disk, initializing [ 1843.967747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1843.999420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.003328] Lustre: Skipped 1 previous similar message [ 1845.958262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1845.961313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1846.001204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1846.111691] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1849.271343] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1850.602541] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1850.672834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1852.011645] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1852.094591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1856.482249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1856.483543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.485306] LustreError: Skipped 1 previous similar message [ 1856.492381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1856.496319] Lustre: Skipped 1 previous similar message [ 1861.604398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.607748] Lustre: Skipped 1 previous similar message [ 1866.721825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.723944] Lustre: Skipped 1 previous similar message [ 1867.231130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1867.249080] LustreError: 114226:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1867.251175] LustreError: 114226:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1867.289066] Lustre: server umount lustre-OST0000 complete [ 1871.840571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1871.841053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.845915] Lustre: Skipped 1 previous similar message [ 1874.781077] Lustre: server umount lustre-MDT0000 complete [ 1876.020617] LustreError: 112067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746461 with bad export cookie 2279172369168896824 [ 1876.023634] LustreError: 112067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1876.024905] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1876.086591] LustreError: 114629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1876.088290] LustreError: 114629:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1876.138264] Lustre: server umount lustre-MDT0001 complete [ 1879.013749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.190676] LustreError: 115104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.197970] LustreError: 115104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1879.218331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.558167] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1883.327197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1884.624804] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1886.393270] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.645547] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1890.079840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1892.250018] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1894.191708] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1912.098400] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1913.824047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1913.826216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.829880] Lustre: Skipped 1 previous similar message [ 1913.831468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1913.832854] Lustre: Skipped 2 previous similar messages [ 1919.150611] LustreError: 117466:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1919.152275] LustreError: 117466:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1919.187524] Lustre: server umount lustre-OST0000 complete [ 1924.576095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1924.578102] LustreError: Skipped 1 previous similar message [ 1924.579308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.582441] Lustre: Skipped 1 previous similar message [ 1926.612866] Lustre: server umount lustre-MDT0000 complete [ 1927.852259] LustreError: 117669:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746513 with bad export cookie 2279172369168897692 [ 1927.856601] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.972858] Lustre: server umount lustre-MDT0001 complete [ 1931.589651] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1932.639716] Key type lgssc unregistered [ 1932.758385] LNet: 118401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.760961] LNetError: 118401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.771515] LNet: Removed LNI 192.168.203.108@tcp [ 1933.089987] Key type .llcrypt unregistered [ 1933.091149] Key type ._llcrypt unregistered [ 1939.157628] Key type ._llcrypt registered [ 1939.158679] Key type .llcrypt registered [ 1939.193190] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 1945.481398] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1945.801430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.841684] alg: No test for adler32 (adler32-zlib) [ 1946.691121] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1946.772279] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1948.351146] Key type lgssc registered [ 1948.695524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.485788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1953.407798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1955.399059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1957.403129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1959.619588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1959.642350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1959.646434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.725697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1960.737764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1960.775828] Lustre: lustre-MDT0000: new disk, initializing [ 1960.802474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.809265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1962.116129] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1966.656472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1966.695474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1966.739763] Lustre: 121504:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1966.775709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1966.779120] Lustre: Skipped 1 previous similar message [ 1966.823852] Lustre: lustre-MDT0001: new disk, initializing [ 1966.870670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.886139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1966.888635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1968.414636] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1972.743228] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1975.088101] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1978.542948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1978.575836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1978.705620] Lustre: lustre-OST0000: new disk, initializing [ 1978.708697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1978.751778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.730276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1979.738225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1979.796288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1981.564677] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1986.387717] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1989.230729] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1989.386833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1991.991127] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1992.183569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1995.232527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1995.234623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.235175] LustreError: Skipped 1 previous similar message [ 1995.235794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.239499] Lustre: Skipped 1 previous similar message [ 1995.244634] Lustre: Skipped 1 previous similar message [ 1999.341903] LustreError: 124442:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1999.376966] Lustre: server umount lustre-OST0000 complete [ 2002.912055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2002.915132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.918969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.472152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.472576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.722028] LustreError: 124643:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2006.724936] LustreError: 124643:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2006.751525] Lustre: server umount lustre-MDT0000 complete [ 2007.940039] LustreError: 121496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746593 with bad export cookie 13250215962533613501 [ 2007.943365] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.944240] LustreError: 121496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2008.009601] LustreError: 124845:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2008.011282] LustreError: 124845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2008.062820] Lustre: server umount lustre-MDT0001 complete [ 2010.665858] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:23:15 (1764746595) [ 2012.692864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.876925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.209957] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2017.037066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2017.159087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2018.400200] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2020.112644] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2021.303153] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2021.968731] LustreError: 125546:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2021.970662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.971837] LustreError: 125546:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2021.973760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.973764] Lustre: Skipped 1 previous similar message [ 2021.974312] Lustre: Skipped 1 previous similar message [ 2021.977578] LustreError: 125546:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2026.975954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.977313] Lustre: Skipped 2 previous similar messages [ 2028.090090] LustreError: 126857:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2028.092917] LustreError: 126857:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2028.118836] Lustre: server umount lustre-MDT0000 complete [ 2029.260346] LustreError: 126860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746615 with bad export cookie 13250215962533614369 [ 2029.263440] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.264198] LustreError: 126860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2029.371183] Lustre: server umount lustre-MDT0001 complete [ 2031.979957] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:23:37 (1764746617) [ 2034.163550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2049.567612] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2054.687529] LustreError: 127702:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 2080.287303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.385178] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2084.314685] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2085.037055] LustreError: 128366:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2085.039295] LustreError: 128366:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2085.083432] Lustre: server umount lustre-OST0000 complete [ 2087.972093] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:24:32 (1764746672) [ 2090.195552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.341816] LustreError: 129036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.347236] LustreError: 129036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2091.669401] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2094.468901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2095.833402] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2097.516863] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2098.626379] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2100.884214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2101.009422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.012033] Lustre: Skipped 2 previous similar messages [ 2102.947646] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2104.943110] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2120.159764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2120.162260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.167862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.170657] Lustre: Skipped 2 previous similar messages [ 2123.068741] LustreError: 131304:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2123.070418] LustreError: 131304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2123.101611] Lustre: server umount lustre-MDT0000 complete [ 2124.297342] LustreError: 129016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746710 with bad export cookie 13250215962533615006 [ 2124.299623] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2124.300853] LustreError: 129016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2124.409353] Lustre: server umount lustre-MDT0001 complete [ 2135.891377] Lustre: server umount lustre-OST0000 complete [ 2138.613657] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:25:23 (1764746723) [ 2140.811618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.951519] LustreError: 132376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.973519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.318394] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2145.233373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2146.674627] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2148.451381] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2149.686868] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2151.959461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2154.147849] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2156.072348] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2157.536754] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2166.437211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2166.781292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2168.080806] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2168.151814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2171.872237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2171.872339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.874589] LustreError: Skipped 1 previous similar message [ 2171.877017] Lustre: Skipped 2 previous similar messages [ 2171.877467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.883221] Lustre: Skipped 3 previous similar messages [ 2174.829170] LustreError: 135191:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2174.830520] LustreError: 135191:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2174.863088] Lustre: server umount lustre-OST0000 complete [ 2182.228535] Lustre: server umount lustre-MDT0000 complete [ 2183.558385] LustreError: 132356:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746769 with bad export cookie 13250215962533616070 [ 2183.562252] LustreError: 132356:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2183.563178] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2183.681802] Lustre: server umount lustre-MDT0001 complete [ 2186.511187] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:26:11 (1764746771) [ 2188.678740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2204.127749] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2209.247355] LustreError: 136238:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 2234.847616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.850525] Lustre: Skipped 2 previous similar messages [ 2236.795151] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2238.598261] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2241.018226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.170328] LustreError: 136265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.222664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2242.653495] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2245.711735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2247.167277] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2248.952272] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2250.162942] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2251.498297] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2251.569578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2252.864673] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2252.955044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2255.331072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.332033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.335073] Lustre: Skipped 4 previous similar messages [ 2255.337860] Lustre: Skipped 4 previous similar messages [ 2259.773657] LustreError: 139012:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2259.775516] LustreError: 139012:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2259.804108] Lustre: server umount lustre-MDT0000 complete [ 2260.450137] LustreError: 137891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.455894] LustreError: 137891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2261.067132] LustreError: 136246:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746846 with bad export cookie 13250215962533617078 [ 2261.070145] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.071745] LustreError: 136246:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2261.189778] Lustre: server umount lustre-MDT0001 complete [ 2272.737704] Lustre: server umount lustre-OST0000 complete [ 2275.707250] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:27:40 (1764746860) [ 2278.094183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2293.535505] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2298.655396] LustreError: 140059:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 2326.218715] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2328.026249] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2330.400304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2330.553272] LustreError: 140086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.605109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2331.978712] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2335.054866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2336.707206] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2338.561788] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2339.898331] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2342.347452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2342.367037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2342.431950] Lustre: lustre-OST0001: new disk, initializing [ 2342.434269] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2344.181921] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2345.203925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2345.206657] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2345.238467] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2347.237305] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2348.511871] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2348.577017] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2349.762400] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2349.838396] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2350.562974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.563672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.566158] Lustre: Skipped 1 previous similar message [ 2350.569350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.572366] Lustre: Skipped 1 previous similar message [ 2356.522089] LustreError: 143790:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2356.523651] LustreError: 143790:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2356.538192] Lustre: server umount lustre-OST0000 complete [ 2360.801749] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.803770] LustreError: Skipped 1 previous similar message [ 2360.804409] LustreError: 140077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.811591] LustreError: 140077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2372.063125] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2372.118671] Lustre: server umount lustre-OST0001 complete [ 2379.610555] Lustre: server umount lustre-MDT0000 complete [ 2380.951360] LustreError: 142606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746966 with bad export cookie 13250215962533618023 [ 2380.954328] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.956647] LustreError: 142606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2381.106469] Lustre: server umount lustre-MDT0001 complete [ 2383.086341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.408693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2385.727953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2387.088152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2389.369929] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:29:34 (1764746974) [ 2389.980459] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2390.547563] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2391.124771] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:29:36 (1764746976) [ 2393.448542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2393.554346] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2393.566739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2393.647745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.651370] Lustre: Skipped 6 previous similar messages [ 2394.996242] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2397.880550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2397.913223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2397.923182] Lustre: 146028:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2399.393369] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2401.230354] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2402.460163] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2404.814949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2404.887476] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2406.861968] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2407.440109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2408.658932] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2410.095017] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2410.188989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2411.568516] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2411.643132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2412.512979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2412.514958] LustreError: Skipped 2 previous similar messages [ 2414.419536] Lustre: server umount lustre-OST0000 complete [ 2428.195025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2430.042435] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2431.889931] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2432.419769] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.423241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.423388] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2432.429534] Lustre: Skipped 1 previous similar message [ 2433.308692] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2433.374658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2434.585188] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2434.651833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2435.824925] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d6c19c1a000.ost_server_uuid 50 [ 2436.319894] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d6c19c1a000.ost_server_uuid in FULL state after 0 sec [ 2436.381476] LustreError: 147080:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.203.8@tcp) returned error from blocking AST (req@000000008e453293 x1850470859723264 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000089120d7e/0xb7e239371c06eaac lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.8@tcp remote: 0x456160bc19871186 expref: 5 pid: 146035 timeout: 2536 lvb_type: 0 [ 2436.396020] LustreError: lustre-MDT0000: A client on nid 192.168.203.8@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2436.398169] LustreError: 146024:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.8@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000089120d7e/0xb7e239371c06eaac lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.8@tcp remote: 0x456160bc19871186 expref: 6 pid: 146035 timeout: 0 lvb_type: 0 [ 2437.601215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.607206] Lustre: Skipped 7 previous similar messages [ 2443.463630] Lustre: server umount lustre-OST0000 complete [ 2450.772233] Lustre: server umount lustre-MDT0000 complete [ 2451.938179] LustreError: 146020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747037 with bad export cookie 13250215962533619199 [ 2451.941714] LustreError: 146020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2451.941799] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.271130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2466.352769] Lustre: server umount lustre-MDT0001 complete [ 2469.770859] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2470.788618] Key type lgssc unregistered [ 2470.926461] LNet: 151511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.928604] LNetError: 151511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2470.940344] LNet: Removed LNI 192.168.203.108@tcp [ 2471.242264] Key type .llcrypt unregistered [ 2471.243301] Key type ._llcrypt unregistered [ 2479.947497] Key type ._llcrypt registered [ 2479.948341] Key type .llcrypt registered [ 2479.983266] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2480.267023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2480.314341] alg: No test for adler32 (adler32-zlib) [ 2481.153469] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2481.234938] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2482.815182] Key type lgssc registered [ 2483.155207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.694433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.700621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.860688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.224547] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2491.177244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2491.293618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.559262] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2494.302688] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2495.467308] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2497.636967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2497.738794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.516856] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2501.225573] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2512.102191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2518.998974] Lustre: Failing over lustre-MDT0000 [ 2519.106539] LustreError: 155149:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2519.129927] Lustre: server umount lustre-MDT0000 complete [ 2522.079877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2522.082548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.087183] LustreError: 154255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.093570] LustreError: 154255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2522.593071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.593968] LustreError: 154254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.596935] Lustre: Skipped 1 previous similar message [ 2527.712737] LustreError: 152880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.718681] LustreError: 152880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2530.011694] LustreError: 152862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.8@tcp arrived at 1764747115 with bad export cookie 157912466190930821 [ 2530.017802] LustreError: 152862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2532.246163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.281328] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2532.372120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.392159] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2533.652839] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2534.340507] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2534.343276] Lustre: lustre-MDT0000: Denying connection for new client 4413867c-dd08-4bcf-912a-24a981882c82 (at 192.168.203.8@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2537.444125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2542.561378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.562905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.564843] Lustre: Skipped 1 previous similar message [ 2542.568054] Lustre: Skipped 1 previous similar message [ 2547.681314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2547.682964] Lustre: Skipped 1 previous similar message [ 2552.801535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2552.803671] Lustre: Skipped 1 previous similar message [ 2555.359122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2555.380454] LustreError: 156081:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2555.382511] LustreError: 156081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2555.416218] Lustre: server umount lustre-OST0000 complete [ 2556.586267] LustreError: 156286:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2556.588985] Lustre: 155650:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2556.591337] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2556.596949] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2556.601531] LustreError: 155650:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000064bc16e1 x1850471420563072/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2556.606976] LustreError: 155650:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2556.610107] LustreError: 155650:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2556.613234] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 2556.690646] LustreError: 156286:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2556.693245] LustreError: 156286:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2556.721254] Lustre: server umount lustre-MDT0000 complete [ 2557.894216] LustreError: 155270:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747143 with bad export cookie 157912466190931178 [ 2557.897655] LustreError: 152191:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2557.898220] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.898610] LustreError: 155270:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2564.027083] LustreError: 156488:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2564.029020] LustreError: 156488:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2564.074638] Lustre: server umount lustre-MDT0001 complete [ 2567.208353] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2568.138468] Key type lgssc unregistered [ 2568.269380] LNet: 157021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.272184] LNetError: 157021:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.282297] LNet: Removed LNI 192.168.203.108@tcp [ 2568.579850] Key type .llcrypt unregistered [ 2568.581175] Key type ._llcrypt unregistered [ 2576.840848] Key type ._llcrypt registered [ 2576.842018] Key type .llcrypt registered [ 2576.881806] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2577.296625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2577.305458] alg: No test for adler32 (adler32-zlib) [ 2578.161104] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2578.252697] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2579.839156] Key type lgssc registered [ 2580.220289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2582.797830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2582.803499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.959742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.319387] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2588.243056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2588.369112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2589.655938] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2591.494411] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2592.823197] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2595.138486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2595.250721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2597.145606] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2598.956740] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2608.553329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2613.728864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2613.731331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.732855] LustreError: Skipped 1 previous similar message [ 2613.733076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2613.737080] Lustre: Skipped 1 previous similar message [ 2616.879298] LustreError: 160661:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2616.919382] Lustre: server umount lustre-OST0000 complete [ 2618.848473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.849245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2618.852084] Lustre: Skipped 1 previous similar message [ 2618.854096] Lustre: Skipped 1 previous similar message [ 2623.968313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2623.970888] Lustre: Skipped 2 previous similar messages [ 2624.317443] LustreError: 160862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2624.319685] LustreError: 160862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2624.351880] Lustre: server umount lustre-MDT0000 complete [ 2625.635340] LustreError: 158373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747211 with bad export cookie 18312686617681200791 [ 2625.638474] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.640947] LustreError: 158373:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2625.706750] LustreError: 161064:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2625.708559] LustreError: 161064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2625.760468] Lustre: server umount lustre-MDT0001 complete [ 2629.348582] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2630.373260] Key type lgssc unregistered [ 2630.495433] LNet: 161594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2630.497832] LNetError: 161594:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.505339] LNet: Removed LNI 192.168.203.108@tcp [ 2630.796780] Key type .llcrypt unregistered [ 2630.798114] Key type ._llcrypt unregistered [ 2642.752045] Key type ._llcrypt registered [ 2642.753686] Key type .llcrypt registered [ 2642.787814] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2643.115021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2643.122888] alg: No test for adler32 (adler32-zlib) [ 2643.974310] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2644.062605] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2645.647177] Key type lgssc registered [ 2646.008273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2648.721677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2648.727054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.905439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.259206] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2654.266338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2654.397522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2655.786562] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2657.718333] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2658.917300] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2661.169434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2661.282711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.236302] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2665.124309] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2674.602279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2679.776709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2679.778970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.780496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2679.783331] Lustre: Skipped 1 previous similar message [ 2679.786340] Lustre: Skipped 1 previous similar message [ 2683.053263] LustreError: 165523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2683.088686] Lustre: server umount lustre-OST0000 complete [ 2684.896396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.897045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.901579] Lustre: Skipped 1 previous similar message [ 2690.016338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.018755] Lustre: Skipped 2 previous similar messages [ 2690.428133] LustreError: 165725:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2690.430047] LustreError: 165725:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2690.454076] Lustre: server umount lustre-MDT0000 complete [ 2691.682189] LustreError: 163233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747277 with bad export cookie 7229702643441424976 [ 2691.685177] LustreError: 163233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2691.685317] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2691.752166] LustreError: 165926:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2691.753712] LustreError: 165926:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2691.807668] Lustre: server umount lustre-MDT0001 complete [ 2695.417516] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2696.468561] Key type lgssc unregistered [ 2696.597438] LNet: 166458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2696.600653] LNetError: 166458:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2696.607433] LNet: Removed LNI 192.168.203.108@tcp [ 2696.961134] Key type .llcrypt unregistered [ 2696.962988] Key type ._llcrypt unregistered [ 2705.872930] Key type ._llcrypt registered [ 2705.874681] Key type .llcrypt registered [ 2705.911732] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2706.360902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2706.379398] alg: No test for adler32 (adler32-zlib) [ 2707.250838] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2707.340872] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2708.927145] Key type lgssc registered [ 2709.277506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2712.771083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2712.776368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.913241] Lustre: *** cfs_fail_loc=135, val=0*** [ 2713.954901] LustreError: 167939:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2713.956959] LustreError: 167939:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.108@tcp: cfg command failed: rc = -2 [ 2713.959204] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2713.959204] [ 2713.961654] LustreError: MGC192.168.203.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2713.966556] LustreError: 167900:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2713.968828] LustreError: 167900:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2713.971459] LustreError: 167900:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2713.974111] LustreError: 167900:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2714.004676] Lustre: server umount lustre-MDT0000 complete [ 2714.006118] LustreError: 167900:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2716.242288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.400983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.722122] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2720.591311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2720.712776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2722.026642] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2723.839934] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2725.060000] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2725.721479] LustreError: 168340:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2725.724980] LustreError: 168340:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2725.728145] LustreError: 168340:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2725.856371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2725.856578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.857342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.867371] Lustre: Skipped 1 previous similar message [ 2730.976212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.977995] Lustre: Skipped 2 previous similar messages [ 2736.096107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.098035] Lustre: Skipped 1 previous similar message [ 2740.191087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2740.224169] LustreError: 169652:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2740.226772] LustreError: 169652:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2740.249538] Lustre: server umount lustre-MDT0000 complete [ 2741.216389] LustreError: 168311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.222872] LustreError: 168311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2741.503397] LustreError: 169655:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747327 with bad export cookie 10446093564968524692 [ 2741.506760] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2741.509749] LustreError: 169655:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2741.574483] LustreError: 169859:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2741.576193] LustreError: 169859:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2741.637990] Lustre: server umount lustre-MDT0001 complete [ 2745.378773] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2746.398390] Key type lgssc unregistered [ 2746.532303] LNet: 170392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2746.535699] LNetError: 170392:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2746.543344] LNet: Removed LNI 192.168.203.108@tcp [ 2746.855719] Key type .llcrypt unregistered [ 2746.857044] Key type ._llcrypt unregistered [ 2753.979761] Key type ._llcrypt registered [ 2753.980722] Key type .llcrypt registered [ 2754.017277] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2754.112227] Key type .llcrypt unregistered [ 2754.113442] Key type ._llcrypt unregistered [ 2760.931743] Key type ._llcrypt registered [ 2760.932689] Key type .llcrypt registered [ 2760.966392] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2761.393442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2761.399451] alg: No test for adler32 (adler32-zlib) [ 2762.252416] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2762.332614] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2763.911246] Key type lgssc registered [ 2764.261869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2766.756800] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2766.765069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2782.239352] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2787.359183] LustreError: 172457:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 2813.983188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.830108] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2817.611397] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2819.862202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.065161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.093329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2821.487946] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2824.493443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2824.630933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2826.010281] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2827.923607] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.143641] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2830.090209] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2839.010508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.014683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.017029] Lustre: Skipped 1 previous similar message [ 2844.130519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.132136] Lustre: Skipped 1 previous similar message [ 2849.250463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.252482] Lustre: Skipped 1 previous similar message [ 2853.343086] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2853.364973] LustreError: 175226:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2853.397583] Lustre: server umount lustre-OST0000 complete [ 2855.391910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2855.394025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.398247] Lustre: Skipped 1 previous similar message [ 2855.400166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2859.488146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2859.488631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.731387] LustreError: 175429:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2860.733705] LustreError: 175429:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2860.756813] Lustre: server umount lustre-MDT0000 complete [ 2861.987249] LustreError: 172464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747447 with bad export cookie 10443749802778075004 [ 2861.990500] LustreError: 172464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2861.990563] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.053814] LustreError: 175630:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2862.056815] LustreError: 175630:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2862.107193] Lustre: server umount lustre-MDT0001 complete [ 2865.729764] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2866.767161] Key type lgssc unregistered [ 2866.889365] LNet: 176160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.892947] LNetError: 176160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2866.901277] LNet: Removed LNI 192.168.203.108@tcp [ 2867.203556] Key type .llcrypt unregistered [ 2867.204461] Key type ._llcrypt unregistered [ 2876.243101] Key type ._llcrypt registered [ 2876.244048] Key type .llcrypt registered [ 2876.281841] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2876.703359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2876.714589] alg: No test for adler32 (adler32-zlib) [ 2877.567579] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2877.654434] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2879.231109] Key type lgssc registered [ 2879.602855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2882.307213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2882.313877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.470959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.771522] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2887.578355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2887.695215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.990687] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2890.823029] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2892.036313] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2894.267763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2894.379793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.231750] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2898.031372] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2908.709239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2914.863556] Lustre: Failing over lustre-MDT0000 [ 2914.932845] LustreError: 179847:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2914.957725] Lustre: server umount lustre-MDT0000 complete [ 2918.367889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2918.371942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.378162] LustreError: 177531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.382933] LustreError: 177531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2918.882597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.884039] LustreError: 177526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.885958] Lustre: Skipped 1 previous similar message [ 2918.891495] LustreError: 177526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2923.925898] LustreError: 177556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.941565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.980312] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2928.080439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.104357] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2929.046374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2929.493816] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2931.050867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2933.221748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2933.227725] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2933.242867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2934.264616] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2939.873371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2939.876391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.881374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.457693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.458306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2943.461369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.493897] LustreError: 181079:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2944.495758] LustreError: 181079:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2944.533176] Lustre: server umount lustre-OST0000 complete [ 2948.578028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2948.578155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.579812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.579820] Lustre: Skipped 1 previous similar message [ 2948.585952] Lustre: Skipped 1 previous similar message [ 2953.696898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.700367] Lustre: Skipped 2 previous similar messages [ 2958.816224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2958.818018] Lustre: Skipped 1 previous similar message [ 2960.351111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2960.387098] LustreError: 181284:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2960.389681] LustreError: 181284:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2960.414853] Lustre: server umount lustre-MDT0000 complete [ 2961.625162] LustreError: 177512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747547 with bad export cookie 6526737794073584154 [ 2961.627675] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.628528] LustreError: 177512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2961.739354] Lustre: server umount lustre-MDT0001 complete [ 2965.417240] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2966.506401] Key type lgssc unregistered [ 2966.637282] LNet: 182017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2966.640310] LNetError: 182017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2966.653352] LNet: Removed LNI 192.168.203.108@tcp [ 2966.965250] Key type .llcrypt unregistered [ 2966.966426] Key type ._llcrypt unregistered [ 2975.995835] Key type ._llcrypt registered [ 2975.996720] Key type .llcrypt registered [ 2976.030414] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2976.390259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2976.420793] alg: No test for adler32 (adler32-zlib) [ 2977.266186] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2977.340213] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2978.919128] Key type lgssc registered [ 2979.282921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2981.930207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2981.936588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.119639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.463219] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2987.310234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2987.437660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.746689] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2990.557069] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2991.706030] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2993.954771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2994.091236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2996.034403] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2997.809540] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3007.401206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3008.984570] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3015.514930] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3024.576739] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3038.180260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.181741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3038.183765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.184416] Lustre: Skipped 1 previous similar message [ 3042.095200] LustreError: 185841:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3042.134700] Lustre: server umount lustre-OST0000 complete [ 3043.807829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3043.811977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.819127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.821434] Lustre: Skipped 1 previous similar message [ 3048.417480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.418113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.536987] LustreError: 186042:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3049.538920] LustreError: 186042:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3049.566463] Lustre: server umount lustre-MDT0000 complete [ 3050.948303] LustreError: 183366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747636 with bad export cookie 593919987819929473 [ 3050.951574] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3050.954788] LustreError: 183366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3051.029775] LustreError: 186244:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3051.031433] LustreError: 186244:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3051.084139] Lustre: server umount lustre-MDT0001 complete [ 3055.116565] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3056.193456] Key type lgssc unregistered [ 3056.313292] LNet: 186775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3056.315479] LNetError: 186775:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3056.324330] LNet: Removed LNI 192.168.203.108@tcp [ 3056.641095] Key type .llcrypt unregistered [ 3056.642575] Key type ._llcrypt unregistered [ 3065.737688] Key type ._llcrypt registered [ 3065.738672] Key type .llcrypt registered [ 3065.773525] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3066.183571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.194529] alg: No test for adler32 (adler32-zlib) [ 3067.046454] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3067.122561] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3068.695087] Key type lgssc registered [ 3069.025656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3071.594791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3071.601219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.758091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3074.007668] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3077.233071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3077.364952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3078.752616] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3080.804778] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3082.039349] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3084.196778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3084.312438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.001926] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3087.679186] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3097.637564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3099.829669] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3107.937871] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3116.804418] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3122.289939] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3133.410125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.411153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.413271] LustreError: Skipped 1 previous similar message [ 3133.417061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.419524] Lustre: Skipped 1 previous similar message [ 3138.413364] LustreError: 192232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3138.448495] Lustre: server umount lustre-OST0000 complete [ 3143.648854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.649389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.651927] Lustre: Skipped 2 previous similar messages [ 3143.654621] Lustre: Skipped 1 previous similar message [ 3145.794722] LustreError: 192434:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3145.797393] LustreError: 192434:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3145.824997] Lustre: server umount lustre-MDT0000 complete [ 3147.392316] LustreError: 188127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747733 with bad export cookie 11395336598376841341 [ 3147.399715] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.405444] LustreError: 188127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.502162] LustreError: 192637:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3147.505092] LustreError: 192637:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3147.556395] Lustre: server umount lustre-MDT0001 complete [ 3151.365090] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3152.523675] Key type lgssc unregistered [ 3152.659497] LNet: 193167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3152.661881] LNetError: 193167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.669539] LNet: Removed LNI 192.168.203.108@tcp [ 3152.984682] Key type .llcrypt unregistered [ 3152.986980] Key type ._llcrypt unregistered [ 3162.260884] Key type ._llcrypt registered [ 3162.261894] Key type .llcrypt registered [ 3162.303625] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3162.807287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3162.837567] alg: No test for adler32 (adler32-zlib) [ 3163.715750] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3163.814329] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3165.399219] Key type lgssc registered [ 3165.781278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3168.587198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3168.594079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.747898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.126956] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3174.049975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3174.185642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3175.546214] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3177.283581] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3178.581986] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3180.763602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3180.879660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.751559] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3184.504762] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3194.214055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3199.457723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3199.458657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.460087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3199.465611] Lustre: Skipped 1 previous similar message [ 3200.621552] LustreError: 196952:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3200.657313] Lustre: server umount lustre-OST0000 complete [ 3204.577206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.578127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.580806] Lustre: Skipped 1 previous similar message [ 3204.584426] Lustre: Skipped 2 previous similar messages [ 3208.001502] LustreError: 197154:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3208.003312] LustreError: 197154:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3208.032528] Lustre: server umount lustre-MDT0000 complete [ 3209.222056] LustreError: 194518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747795 with bad export cookie 2318567404718681521 [ 3209.225303] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3209.225745] LustreError: 194518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3209.286459] LustreError: 197356:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3209.288736] LustreError: 197356:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3209.335406] Lustre: server umount lustre-MDT0001 complete [ 3213.011851] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3214.078852] Key type lgssc unregistered [ 3214.219378] LNet: 197887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3214.221805] LNetError: 197887:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3214.231278] LNet: Removed LNI 192.168.203.108@tcp [ 3214.554985] Key type .llcrypt unregistered [ 3214.556801] Key type ._llcrypt unregistered [ 3222.621096] Key type ._llcrypt registered [ 3222.623110] Key type .llcrypt registered [ 3222.673566] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3223.111427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3223.127383] alg: No test for adler32 (adler32-zlib) [ 3223.989987] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3224.074186] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3225.663235] Key type lgssc registered [ 3226.197846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3229.332820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3229.338261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.503078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.881981] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3234.945249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3235.070482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3236.344571] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3238.159210] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3239.411771] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3242.052498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3242.169554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.022929] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3245.714827] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3255.465963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3260.897225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3260.897398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.899057] LustreError: Skipped 1 previous similar message [ 3260.902730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3260.905912] Lustre: Skipped 1 previous similar message [ 3261.486097] LustreError: 201528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3261.521792] Lustre: server umount lustre-OST0000 complete [ 3266.016886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.017197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3266.021456] Lustre: Skipped 1 previous similar message [ 3266.026377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.028862] Lustre: Skipped 1 previous similar message [ 3268.861759] LustreError: 201730:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3268.863597] LustreError: 201730:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3268.892049] Lustre: server umount lustre-MDT0000 complete [ 3270.180470] LustreError: 199093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747855 with bad export cookie 9388611130688822520 [ 3270.185281] LustreError: 199093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3270.187247] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3270.276496] LustreError: 201931:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3270.279047] LustreError: 201931:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3270.347620] Lustre: server umount lustre-MDT0001 complete [ 3274.069473] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3275.023039] Key type lgssc unregistered [ 3275.148391] LNet: 202461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3275.151313] LNetError: 202461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3275.159305] LNet: Removed LNI 192.168.203.108@tcp [ 3275.430358] Key type .llcrypt unregistered [ 3275.431394] Key type ._llcrypt unregistered [ 3283.434715] Key type ._llcrypt registered [ 3283.435809] Key type .llcrypt registered [ 3283.472493] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3283.769650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3283.780407] alg: No test for adler32 (adler32-zlib) [ 3284.647796] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3284.730949] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3286.311167] Key type lgssc registered [ 3286.749661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3289.778595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3289.791720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.972275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.331267] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3296.546746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3296.725141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3298.367571] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3301.178465] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3302.927789] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3306.005970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3306.161667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.369129] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3310.201908] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3315.367112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3317.890816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3317.913832] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3317.999878] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3320.189972] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3322.436382] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3323.175853] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3333.543931] Lustre: Permanently deactivating lustre-OST0001 [ 3333.550753] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3342.334690] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3344.669567] Lustre: Permanently reactivating lustre-OST0001 [ 3344.674225] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3344.676161] Lustre: Skipped 2 previous similar messages [ 3352.056502] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.060199] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3352.062744] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3352.067326] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3352.080196] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3354.599874] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.601799] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3354.602895] Lustre: Skipped 2 previous similar messages [ 3354.607940] Lustre: Skipped 1 previous similar message [ 3359.213096] LustreError: 207108:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3359.228595] Lustre: server umount lustre-OST0001 complete [ 3359.715447] LustreError: 205284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.720970] LustreError: 205284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3361.759830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.761884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.765487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3363.807749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.809738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.813418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.835544] LustreError: 205284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.839644] LustreError: 205284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3366.445098] LustreError: 207310:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3366.446769] LustreError: 207310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3366.478321] Lustre: server umount lustre-OST0000 complete [ 3368.416309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3368.418862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.422339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.426528] Lustre: Skipped 2 previous similar messages [ 3375.072083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.074054] Lustre: Skipped 3 previous similar messages [ 3382.239129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3382.294123] LustreError: 207512:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3382.297814] LustreError: 207512:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3382.332485] Lustre: server umount lustre-MDT0000 complete [ 3383.556057] LustreError: 203664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747969 with bad export cookie 7372229227449446416 [ 3383.559253] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3383.559607] LustreError: 203664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3383.671261] Lustre: server umount lustre-MDT0001 complete [ 3387.574948] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3388.587479] Key type lgssc unregistered [ 3388.716436] LNet: 208247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3388.718870] LNetError: 208247:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3388.727524] LNet: Removed LNI 192.168.203.108@tcp [ 3389.047198] Key type .llcrypt unregistered [ 3389.048206] Key type ._llcrypt unregistered [ 3401.109792] Key type ._llcrypt registered [ 3401.110649] Key type .llcrypt registered [ 3401.148174] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3407.983665] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3408.308657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3408.341048] alg: No test for adler32 (adler32-zlib) [ 3409.203816] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3409.290057] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3410.871153] Key type lgssc registered [ 3411.229920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3414.297991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3416.410895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3418.773298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3421.379924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3423.928700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3423.950638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3423.956076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.037666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3425.048274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3425.086721] Lustre: lustre-MDT0000: new disk, initializing [ 3425.110135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.116179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3426.320673] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3430.623452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3430.646062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3430.676793] Lustre: 212164:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3430.697843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3430.699411] Lustre: Skipped 1 previous similar message [ 3430.736055] Lustre: lustre-MDT0001: new disk, initializing [ 3430.764914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3430.775432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3430.778232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3432.005638] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3434.985815] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3436.120768] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3438.322567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3438.349842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3438.441065] Lustre: lustre-OST0000: new disk, initializing [ 3438.442719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3438.469748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.203342] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3441.077167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3441.079697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3441.109305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3443.225206] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3444.560570] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3444.632523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3445.878780] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3445.939023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3451.362372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.362926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3451.365971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3451.367153] LustreError: Skipped 1 previous similar message [ 3451.370723] Lustre: Skipped 1 previous similar message [ 3452.589110] LustreError: 215104:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3452.624805] Lustre: server umount lustre-OST0000 complete [ 3456.480848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.480908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3456.482282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3456.483933] Lustre: Skipped 2 previous similar messages [ 3461.600083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3461.601552] Lustre: Skipped 2 previous similar messages [ 3466.720140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.721841] Lustre: Skipped 1 previous similar message [ 3468.255089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3468.285569] LustreError: 215305:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3468.287093] LustreError: 215305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3468.316652] Lustre: server umount lustre-MDT0000 complete [ 3469.511257] LustreError: 213910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748055 with bad export cookie 7591568483310744486 [ 3469.513779] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.515838] LustreError: 213910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3469.589708] LustreError: 215508:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3469.591471] LustreError: 215508:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3469.648396] Lustre: server umount lustre-MDT0001 complete [ 3472.274157] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 02:47:37 (1764748057) [ 3474.378040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.533331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.680541] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3478.706270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3478.826750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3480.070734] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3482.055869] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3483.737818] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3486.883718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3487.057482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.315370] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3491.311101] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3502.567049] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3507.079358] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3515.742538] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3522.449104] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3532.330611] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3550.898321] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3550.903283] Lustre: Skipped 1 previous similar message [ 3588.800836] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3588.805436] Lustre: Skipped 4 previous similar messages [ 3658.275484] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3658.277632] Lustre: Skipped 8 previous similar messages [ 3668.451585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3668.451748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.452689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3668.453667] LustreError: Skipped 1 previous similar message [ 3668.464780] Lustre: Skipped 1 previous similar message [ 3674.352145] LustreError: 219669:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3674.354862] LustreError: 219669:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3674.394858] Lustre: server umount lustre-OST0000 complete [ 3678.689197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3678.689397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.689852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3678.689857] Lustre: Skipped 3 previous similar messages [ 3678.699456] Lustre: Skipped 1 previous similar message [ 3681.727306] LustreError: 219874:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3681.728890] LustreError: 219874:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3681.754454] Lustre: server umount lustre-MDT0000 complete [ 3683.212937] LustreError: 216159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748269 with bad export cookie 7591568483310745347 [ 3683.219019] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.219588] LustreError: 216159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3683.361852] Lustre: server umount lustre-MDT0001 complete [ 3687.170421] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3688.413658] Key type lgssc unregistered [ 3688.564720] LNet: 220607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3688.569272] LNetError: 220607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3688.577699] LNet: Removed LNI 192.168.203.108@tcp [ 3689.040378] Key type .llcrypt unregistered [ 3689.042350] Key type ._llcrypt unregistered [ 3699.148685] Key type ._llcrypt registered [ 3699.149626] Key type .llcrypt registered [ 3699.184069] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3699.479809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3699.513598] alg: No test for adler32 (adler32-zlib) [ 3700.370695] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3700.450583] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3702.031232] Key type lgssc registered [ 3702.392643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3705.156082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3705.161741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.345230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.532600] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3710.967930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3711.107425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3712.739316] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3714.962669] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3716.186402] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3718.863910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3719.013223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.865312] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3722.651255] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3741.667787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3741.668570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.675151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3741.676534] Lustre: Skipped 1 previous similar message [ 3744.826762] LustreError: 224398:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3744.880088] Lustre: server umount lustre-OST0000 complete [ 3746.786260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.787538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.792781] Lustre: Skipped 2 previous similar messages [ 3751.904212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3751.906354] Lustre: Skipped 2 previous similar messages [ 3752.766116] LustreError: 224603:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3752.768660] LustreError: 224603:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3752.795464] Lustre: server umount lustre-MDT0000 complete [ 3754.049736] LustreError: 221959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748339 with bad export cookie 5411126520707565783 [ 3754.055363] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3754.056546] LustreError: 221959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3754.145076] LustreError: 224805:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3754.147187] LustreError: 224805:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3754.194280] Lustre: server umount lustre-MDT0001 complete [ 3758.455255] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3759.786094] Key type lgssc unregistered [ 3759.942774] LNet: 225336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3759.947581] LNetError: 225336:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3759.960557] LNet: Removed LNI 192.168.203.108@tcp [ 3760.295215] Key type .llcrypt unregistered [ 3760.296809] Key type ._llcrypt unregistered [ 3768.521780] Key type ._llcrypt registered [ 3768.523387] Key type .llcrypt registered [ 3768.589335] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3768.749471] Key type .llcrypt unregistered [ 3768.751169] Key type ._llcrypt unregistered [ 3778.320663] Key type ._llcrypt registered [ 3778.322104] Key type .llcrypt registered [ 3778.358878] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3785.606772] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3785.890103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3785.905052] alg: No test for adler32 (adler32-zlib) [ 3786.752716] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3786.836297] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3788.423193] Key type lgssc registered [ 3789.017246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3792.904365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3795.796334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3798.200682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3801.212874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3801.831177] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3663 sec ========== 02:53:06 (1764748386) [ 3802.525366] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:53:07 (1764748387) === [ 3804.015531] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:53:08 (1764748388) === [ 3819.402413] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3820.673398] Key type lgssc unregistered [ 3820.826794] LNet: 231414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3820.831417] LNetError: 231414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3820.842739] LNet: Removed LNI 192.168.203.108@tcp [ 3821.349827] Key type .llcrypt unregistered [ 3821.351864] Key type ._llcrypt unregistered