[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 571205485 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003359] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008285] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010018] pid_max: default: 32768 minimum: 301 [ 0.012083] LSM: Security Framework initializing [ 0.013045] Yama: becoming mindful. [ 0.014037] SELinux: Initializing. [ 0.015071] *** VALIDATE selinux *** [ 0.021040] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026540] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028176] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030040] *** VALIDATE tmpfs *** [ 0.032424] *** VALIDATE proc *** [ 0.034019] *** VALIDATE cgroup *** [ 0.035000] *** VALIDATE cgroup2 *** [ 0.036214] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037057] Spectre V2 : User space: Vulnerable [ 0.038000] Speculative Store Bypass: Vulnerable [ 0.041255] debug: unmapping init [mem 0xffffffff85659000-0xffffffff85660fff] [ 0.043992] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044509] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046025] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049012] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051007] ... event mask: 000000070000000f [ 0.052340] rcu: Hierarchical SRCU implementation. [ 0.054762] smp: Bringing up secondary CPUs ... [ 0.055483] x86: Booting SMP configuration: [ 0.056013] .... node #0, CPUs: #1 #2 #3 [ 0.071358] smp: Brought up 1 node, 4 CPUs [ 0.073007] smpboot: Max logical packages: 1 [ 0.074020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106570] node 0 deferred pages initialised in 29ms [ 0.109232] devtmpfs: initialized [ 0.110286] x86/mm: Memory block size: 128MB [ 0.111924] gcov: version magic: 0x41383552 [ 0.113310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.117133] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.118401] pinctrl core: initialized pinctrl subsystem [ 0.121214] [ 0.122013] ************************************************************* [ 0.125012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127009] ** ** [ 0.128006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.130010] ** ** [ 0.132011] ** This means that this kernel is built to expose internal ** [ 0.135010] ** IOMMU data structures, which may compromise security on ** [ 0.137010] ** your system. ** [ 0.140011] ** ** [ 0.142008] ** If you see this message and you are not debugging the ** [ 0.143008] ** kernel, report this immediately to your vendor! ** [ 0.147011] ** ** [ 0.154013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159009] ************************************************************* [ 0.163136] NET: Registered protocol family 16 [ 0.164393] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171030] cpuidle: using governor menu [ 0.177348] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180701] PCI: Using configuration type 1 for base access [ 0.183123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193304] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200653] cryptd: max_cpu_qlen set to 1000 [ 0.203243] ACPI: Added _OSI(Module Device) [ 0.205012] ACPI: Added _OSI(Processor Device) [ 0.207010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212910] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220237] ACPI: Interpreter enabled [ 0.221694] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223012] ACPI: Using IOAPIC for interrupt routing [ 0.225101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228451] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239381] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244185] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250315] acpiphp: Slot [2] registered [ 0.252110] acpiphp: Slot [3] registered [ 0.253065] acpiphp: Slot [4] registered [ 0.255072] acpiphp: Slot [5] registered [ 0.256103] acpiphp: Slot [6] registered [ 0.258683] acpiphp: Slot [7] registered [ 0.260067] acpiphp: Slot [8] registered [ 0.261754] acpiphp: Slot [9] registered [ 0.263082] acpiphp: Slot [10] registered [ 0.264090] acpiphp: Slot [11] registered [ 0.266083] acpiphp: Slot [12] registered [ 0.267066] acpiphp: Slot [13] registered [ 0.269060] acpiphp: Slot [14] registered [ 0.270606] acpiphp: Slot [15] registered [ 0.272063] acpiphp: Slot [16] registered [ 0.274109] acpiphp: Slot [17] registered [ 0.275059] acpiphp: Slot [18] registered [ 0.277015] acpiphp: Slot [19] registered [ 0.278073] acpiphp: Slot [20] registered [ 0.279072] acpiphp: Slot [21] registered [ 0.281082] acpiphp: Slot [22] registered [ 0.282067] acpiphp: Slot [23] registered [ 0.284070] acpiphp: Slot [24] registered [ 0.286150] acpiphp: Slot [25] registered [ 0.287064] acpiphp: Slot [26] registered [ 0.289068] acpiphp: Slot [27] registered [ 0.291071] acpiphp: Slot [28] registered [ 0.292105] acpiphp: Slot [29] registered [ 0.293073] acpiphp: Slot [30] registered [ 0.295095] acpiphp: Slot [31] registered [ 0.296056] PCI host bridge to bus 0000:00 [ 0.298014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.300017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.303018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305569] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.308018] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.311017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317361] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.321500] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.330742] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.353428] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355781] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.361613] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.381016] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.385012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.390434] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407354] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.466025] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.479440] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.508014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532016] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.547037] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.553016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.564017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.590018] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.602355] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.609000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.615018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.635015] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.647000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.656015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.664019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.693078] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.709000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.715016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.727018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.751019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.775000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.776141] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.785382] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.790381] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.792232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.800118] iommu: Default domain type: Passthrough [ 0.804386] SCSI subsystem initialized [ 0.805117] ACPI: bus type USB registered [ 0.807081] usbcore: registered new interface driver usbfs [ 0.808055] usbcore: registered new interface driver hub [ 0.810077] usbcore: registered new device driver usb [ 0.812122] pps_core: LinuxPPS API ver. 1 registered [ 0.813007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.816064] PTP clock support registered [ 0.819176] EDAC MC: Ver: 3.0.0 [ 0.821110] PCI: Using ACPI for IRQ routing [ 0.822517] NetLabel: Initializing [ 0.824012] NetLabel: domain hash size = 128 [ 0.825006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.830098] NetLabel: unlabeled traffic allowed by default [ 0.833065] vgaarb: loaded [ 0.834232] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.835000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.837700] clocksource: Switched to clocksource kvm-clock [ 0.945681] VFS: Disk quotas dquot_6.6.0 [ 0.947299] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.949854] *** VALIDATE ramfs *** [ 0.951226] *** VALIDATE hugetlbfs *** [ 0.952871] pnp: PnP ACPI init [ 0.955309] pnp: PnP ACPI: found 6 devices [ 0.970620] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.973520] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.975913] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.978421] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.980904] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.983192] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.986198] NET: Registered protocol family 2 [ 0.988880] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.994048] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.998585] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.004036] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.007734] TCP: Hash tables configured (established 65536 bind 65536) [ 1.010586] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.013353] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016799] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.019374] NET: Registered protocol family 1 [ 1.022738] RPC: Registered named UNIX socket transport module. [ 1.025803] RPC: Registered udp transport module. [ 1.027856] RPC: Registered tcp transport module. [ 1.029680] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.032187] NET: Registered protocol family 44 [ 1.034509] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.036910] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.039139] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.041512] PCI: CLS 0 bytes, default 64 [ 1.043887] Unpacking initramfs... [ 2.854844] debug: unmapping init [mem 0xffff99bafcc54000-0xffff99bafffbffff] [ 2.860140] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.861973] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.864662] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.670305] Initialise system trusted keyrings [ 3.671670] Key type blacklist registered [ 3.674536] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.683538] zbud: loaded [ 3.687131] *** VALIDATE nfs *** [ 3.688232] *** VALIDATE nfs4 *** [ 3.689843] pstore: using deflate compression [ 3.692891] Platform Keyring initialized [ 3.852386] NET: Registered protocol family 38 [ 3.853891] Key type asymmetric registered [ 3.855455] Asymmetric key parser 'x509' registered [ 3.856873] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.859832] io scheduler mq-deadline registered [ 3.861620] io scheduler kyber registered [ 3.863210] io scheduler bfq registered [ 3.867601] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.870914] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.875303] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.877910] ACPI: Power Button [PWRF] [ 3.977668] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.080216] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.413985] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.539358] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.795758] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.831973] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.884066] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.898332] Non-volatile memory driver v1.3 [ 4.899574] Linux agpgart interface v0.103 [ 4.936482] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 4.941609] vda: detected capacity change from 0 to 68530176 [ 4.958448] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.961437] vdb: detected capacity change from 0 to 1073741824 [ 4.987501] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.990522] vdc: detected capacity change from 0 to 2621440000 [ 5.027844] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.031671] vdd: detected capacity change from 0 to 2621440000 [ 5.048797] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.055906] vde: detected capacity change from 0 to 4294967296 [ 5.090283] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.092944] vdf: detected capacity change from 0 to 4294967296 [ 5.102689] libphy: Fixed MDIO Bus: probed [ 5.113386] usbcore: registered new interface driver usbserial_generic [ 5.116970] usbserial: USB Serial support registered for generic [ 5.122277] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.128605] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.132774] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.136910] mousedev: PS/2 mouse device common for all mice [ 5.144092] rtc_cmos 00:05: RTC can wake from S4 [ 5.147979] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.150051] rtc_cmos 00:05: registered as rtc0 [ 5.154986] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.157549] intel_pstate: CPU model not supported [ 5.164913] hid: raw HID events driver (C) Jiri Kosina [ 5.167067] usbcore: registered new interface driver usbhid [ 5.168694] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.168794] usbhid: USB HID core driver [ 5.168941] drop_monitor: Initializing network drop monitor service [ 5.180172] Initializing XFRM netlink socket [ 5.195186] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.199686] NET: Registered protocol family 10 [ 5.216492] Segment Routing with IPv6 [ 5.218166] NET: Registered protocol family 17 [ 5.220716] mpls_gso: MPLS GSO support [ 5.236123] RAS: Correctable Errors collector initialized. [ 5.241877] AVX version of gcm_enc/dec engaged. [ 5.247157] AES CTR mode by8 optimization enabled [ 5.390402] sched_clock: Marking stable (5390317073, 0)->(6441253975, -1050936902) [ 5.394286] registered taskstats version 1 [ 5.396981] Loading compiled-in X.509 certificates [ 5.399590] zswap: loaded using pool lzo/zbud [ 5.429794] Key type big_key registered [ 5.457665] Key type encrypted registered [ 5.459244] ima: No TPM chip found, activating TPM-bypass! [ 5.461238] ima: Allocated hash algorithm: sha1 [ 5.462605] ima: No architecture policies found [ 5.464014] evm: Initialising EVM extended attributes: [ 5.466150] evm: security.selinux [ 5.467566] evm: security.ima [ 5.468600] evm: security.capability [ 5.469805] evm: HMAC attrs: 0x1 [ 5.481515] rtc_cmos 00:05: setting system clock to 2025-11-21 12:09:08 UTC (1763726948) [ 5.496149] debug: unmapping init [mem 0xffffffff86603000-0xffffffff867fffff] [ 5.503322] debug: unmapping init [mem 0xffffffff85382000-0xffffffff85658fff] [ 5.521150] Write protecting the kernel read-only data: 28672k [ 5.525479] debug: unmapping init [mem 0xffffffff83a03000-0xffffffff83bfffff] [ 5.528542] debug: unmapping init [mem 0xffffffff84314000-0xffffffff843fffff] [ 5.603242] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.620676] systemd[1]: Detected virtualization kvm. [ 5.624373] systemd[1]: Detected architecture x86-64. [ 5.627413] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.659582] systemd[1]: No hostname configured. [ 5.661454] systemd[1]: Set hostname to . [ 5.665578] random: systemd: uninitialized urandom read (16 bytes read) [ 5.672994] systemd[1]: Initializing machine ID from random generator. [ 5.922904] random: systemd: uninitialized urandom read (16 bytes read) [ 5.925507] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.932809] random: systemd: uninitialized urandom read (16 bytes read) [ 5.934741] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.944537] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.580964] device-mapper: uevent: version 1.0.3 [ 7.582855] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.710535] virtio_net virtio0 ens2: renamed from eth0 [ 8.846830] random: fast init done [ 9.418579] scsi host0: ata_piix [ 9.449942] scsi host1: ata_piix [ 9.451113] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.452888] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.984548] random: crng init done [ 13.985591] random: 7 urandom warning(s) missed due to ratelimiting [ 14.112652] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.894787] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.792756] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.271190] SELinux: Disabled at runtime. [ 18.343756] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.352294] systemd[1]: Detected virtualization kvm. [ 18.354356] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.373553] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.377638] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.384668] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.388714] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.391895] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.422976] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.445543] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 20.368023] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 20.976051] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.785671] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.816120] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.082499] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.108120] EDAC sbridge: Ver: 1.1.2 [ 26.052636] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 26.711948] NFS: Registering the id_resolver key type [ 26.714287] Key type id_resolver registered [ 26.716693] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 51.354541] libcfs: loading out-of-tree module taints kernel. [ 51.364430] Key type ._llcrypt registered [ 51.365501] Key type .llcrypt registered [ 51.398760] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 58.217731] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 58.734504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.741039] alg: No test for adler32 (adler32-zlib) [ 59.675197] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 59.924299] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 61.527198] Key type lgssc registered [ 62.092992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.448460] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 68.779932] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.519991] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.096285] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.731350] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.038115] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 86.638048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.658583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 86.666728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.769693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 87.782780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 87.827674] Lustre: lustre-MDT0000: new disk, initializing [ 87.858258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.866467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 89.321525] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.772927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.800329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 94.828068] Lustre: 6487:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 94.839689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 94.842420] Lustre: Skipped 1 previous similar message [ 94.879088] Lustre: lustre-MDT0001: new disk, initializing [ 94.901431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 94.910704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 94.914653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 96.267988] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.523729] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 101.665219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.690451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 101.775079] Lustre: lustre-OST0000: new disk, initializing [ 101.776822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 101.794156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 103.617619] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.912953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 103.915594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 103.939036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 108.676587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.704439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 108.744741] Lustre: lustre-OST0001: new disk, initializing [ 108.746966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 108.764439] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 110.585574] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.274653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 111.277764] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 111.294235] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 116.555839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.292202] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 130.150694] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 131.585819] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 133.132045] Lustre: DEBUG MARKER: Client: 2.16.61.40 [ 134.012375] Lustre: DEBUG MARKER: MDS: 2.16.61.40 [ 134.919289] Lustre: DEBUG MARKER: OSS: 2.16.61.40 [ 135.449242] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Nov 21 07:11:17 EST 2025 [ 141.488542] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 142.012001] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 156.639568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 156.643156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.649693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.664063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.664091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 157.666252] Lustre: Skipped 1 previous similar message [ 157.673280] Lustre: Skipped 1 previous similar message [ 159.866870] LustreError: 12189:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 159.894386] Lustre: server umount lustre-MDT0000 complete [ 161.073058] LustreError: 7420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727104 with bad export cookie 1243457717255003279 [ 161.074949] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 161.077856] LustreError: 7420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 162.783713] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 162.784155] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 162.788038] Lustre: Skipped 2 previous similar messages [ 162.790311] Lustre: Skipped 2 previous similar messages [ 166.815835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 166.818468] Lustre: Skipped 2 previous similar messages [ 167.224989] LustreError: 12391:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 167.226929] LustreError: 12391:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 167.265658] Lustre: server umount lustre-MDT0001 complete [ 168.601132] LustreError: 12592:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 168.603656] LustreError: 12592:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 168.623257] Lustre: server umount lustre-OST0000 complete [ 169.952056] Lustre: server umount lustre-OST0001 complete [ 172.062019] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 175.212866] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 178.687595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.865140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.061169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.513838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.039111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.059458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.147460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 188.160864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 188.201659] Lustre: lustre-MDT0000: new disk, initializing [ 188.230582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.238754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.588047] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 194.123503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.154699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 194.198488] Lustre: 14970:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.319764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.798189] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 199.030622] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 200.339324] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 202.787041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.813423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 202.923410] Lustre: lustre-OST0000: new disk, initializing [ 202.924630] Lustre: Skipped 1 previous similar message [ 202.926621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.929244] Lustre: Skipped 2 previous similar messages [ 202.973073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.975938] Lustre: Skipped 1 previous similar message [ 204.674553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.678127] Lustre: Skipped 1 previous similar message [ 204.680769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.713347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.926040] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 208.162905] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 209.623613] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 209.703723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 211.071764] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 211.154848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 215.010767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 215.012217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.012907] LustreError: Skipped 1 previous similar message [ 215.015175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 215.017474] Lustre: Skipped 1 previous similar message [ 225.250893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 225.253017] Lustre: Skipped 4 previous similar messages [ 226.271155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 226.290189] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 226.292935] LustreError: 17909:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 226.329029] Lustre: server umount lustre-OST0000 complete [ 230.369832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 230.370094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.378483] Lustre: Skipped 1 previous similar message [ 233.731173] LustreError: 18111:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 233.734095] LustreError: 18111:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 233.765239] Lustre: server umount lustre-MDT0000 complete [ 234.995630] LustreError: 16717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727178 with bad export cookie 1243457717255004798 [ 234.998070] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 235.001603] LustreError: 16717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 235.113718] Lustre: server umount lustre-MDT0001 complete [ 236.247493] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 07:12:58 (1763727178) [ 238.542351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.707614] LustreError: 18839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 238.715322] LustreError: 18839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 238.736965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.046179] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 243.069088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 244.633334] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 246.458456] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 247.702945] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 249.907724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 251.973588] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 253.768571] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 273.888399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 273.892642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.899160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 273.901818] Lustre: Skipped 2 previous similar messages [ 278.190221] LustreError: 21104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 278.193414] LustreError: 21104:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 278.235078] Lustre: server umount lustre-OST0000 complete [ 285.666454] Lustre: server umount lustre-MDT0000 complete [ 287.061461] LustreError: 19557:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727230 with bad export cookie 1243457717255005680 [ 287.062450] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 287.065587] LustreError: 19557:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 287.199287] Lustre: server umount lustre-MDT0001 complete [ 291.184113] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 292.363679] Key type lgssc unregistered [ 292.511474] LNet: 22040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.516560] LNetError: 22040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 292.531538] LNet: Removed LNI 192.168.202.159@tcp [ 292.923164] Key type .llcrypt unregistered [ 292.924725] Key type ._llcrypt unregistered [ 302.313464] Key type ._llcrypt registered [ 302.314313] Key type .llcrypt registered [ 302.349395] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 302.657212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 302.680783] alg: No test for adler32 (adler32-zlib) [ 303.574224] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 303.663837] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 305.255662] Key type lgssc registered [ 305.683470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 308.815188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 308.823340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 310.020230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 311.407743] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 314.646838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 314.769475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 316.148858] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 318.004316] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 319.310588] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 321.921576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 322.044059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 324.372885] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 326.373045] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 336.421956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 344.544275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 344.548227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.555955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 346.593292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.595138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 346.599770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 346.601089] Lustre: Skipped 1 previous similar message [ 351.713418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 351.715008] Lustre: Skipped 1 previous similar message [ 356.319144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 356.337732] LustreError: 26121:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 356.382443] Lustre: server umount lustre-OST0000 complete [ 360.928234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 360.931291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.936929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.072980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.077328] Lustre: Skipped 3 previous similar messages [ 372.191114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 372.223110] LustreError: 26326:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 372.225826] LustreError: 26326:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 372.253383] Lustre: server umount lustre-MDT0000 complete [ 373.456590] LustreError: 23390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727316 with bad export cookie 16814000867963724974 [ 373.459227] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.460628] LustreError: 23390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 373.534664] LustreError: 26530:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 373.537377] LustreError: 26530:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 373.596313] Lustre: server umount lustre-MDT0001 complete [ 377.158395] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 378.263037] Key type lgssc unregistered [ 378.390758] LNet: 27061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.393306] LNetError: 27061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.404535] LNet: Removed LNI 192.168.202.159@tcp [ 378.751085] Key type .llcrypt unregistered [ 378.752244] Key type ._llcrypt unregistered [ 387.420447] Key type ._llcrypt registered [ 387.421472] Key type .llcrypt registered [ 387.456607] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 387.816897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 387.826716] alg: No test for adler32 (adler32-zlib) [ 388.696074] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 388.775526] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 390.359216] Key type lgssc registered [ 390.757785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 393.505606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 393.513246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.675849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.909528] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 398.803984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 398.923363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.156362] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 401.915328] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 403.100599] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 407.676737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 407.798195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.657141] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 411.421520] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 419.046296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 424.418413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.419875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.421011] LustreError: Skipped 1 previous similar message [ 424.425686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.428472] Lustre: Skipped 1 previous similar message [ 425.198101] LustreError: 31043:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 425.234950] Lustre: server umount lustre-OST0000 complete [ 428.668957] LustreError: 31244:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 428.671511] LustreError: 31244:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 428.705345] Lustre: server umount lustre-MDT0000 complete [ 429.538161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.538751] LustreError: 29154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.543395] Lustre: Skipped 2 previous similar messages [ 429.543761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 429.551692] LustreError: 29154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 429.980021] LustreError: 30944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727372 with bad export cookie 5499406438323626989 [ 429.982890] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 429.985240] LustreError: 30944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 430.048192] LustreError: 31444:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 430.051513] LustreError: 31444:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 430.103593] Lustre: server umount lustre-MDT0001 complete [ 433.717337] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 434.746803] Key type lgssc unregistered [ 434.876362] LNet: 31977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.879379] LNetError: 31977:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 434.890322] LNet: Removed LNI 192.168.202.159@tcp [ 435.171497] Key type .llcrypt unregistered [ 435.172594] Key type ._llcrypt unregistered [ 443.493662] Key type ._llcrypt registered [ 443.494968] Key type .llcrypt registered [ 443.528285] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 443.917896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 443.925394] alg: No test for adler32 (adler32-zlib) [ 444.774475] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 444.851775] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 446.423171] Key type lgssc registered [ 446.758234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 449.346751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 449.352987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.500358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.864920] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 454.622660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 454.733105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 455.944597] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 457.644506] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 458.808908] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 460.965439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 461.060556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.839091] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 464.591653] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 475.429546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 484.832042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.835730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.839723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.857566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.857631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.857685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.859977] Lustre: Skipped 1 previous similar message [ 488.877641] LustreError: 35618:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 488.914953] Lustre: server umount lustre-OST0000 complete [ 490.463798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 490.465816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.470400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 496.096799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 496.099975] Lustre: Skipped 2 previous similar messages [ 496.321100] LustreError: 35819:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 496.323727] LustreError: 35819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 496.346848] Lustre: server umount lustre-MDT0000 complete [ 497.615128] LustreError: 33329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727440 with bad export cookie 1166303760699826329 [ 497.619353] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.620766] LustreError: 33329:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 497.689293] LustreError: 36020:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 497.691502] LustreError: 36020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 497.746086] Lustre: server umount lustre-MDT0001 complete [ 501.346039] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 502.421662] Key type lgssc unregistered [ 502.553403] LNet: 36550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.556800] LNetError: 36550:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 502.568329] LNet: Removed LNI 192.168.202.159@tcp [ 502.871845] Key type .llcrypt unregistered [ 502.873417] Key type ._llcrypt unregistered [ 511.991527] Key type ._llcrypt registered [ 511.993104] Key type .llcrypt registered [ 512.031705] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 512.359057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 512.370282] alg: No test for adler32 (adler32-zlib) [ 513.235019] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 513.329254] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 514.911157] Key type lgssc registered [ 515.308532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 518.129062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 518.134740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.297319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.673072] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 523.577988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 523.706078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 525.123456] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 527.052222] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 528.336752] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 530.771488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 530.905441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.896731] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 534.847191] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 544.233250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 549.346262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.347540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.348484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.348670] LustreError: Skipped 1 previous similar message [ 549.352339] Lustre: Skipped 1 previous similar message [ 550.285573] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 550.287960] Lustre: Skipped 1 previous similar message [ 553.464114] LustreError: 40186:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 553.507681] Lustre: server umount lustre-OST0000 complete [ 566.598481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.603609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 569.696805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 569.699786] Lustre: Skipped 2 previous similar messages [ 572.736839] LustreError: 40562:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 572.739062] LustreError: 40562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 572.763442] Lustre: server umount lustre-MDT0000 complete [ 574.055545] LustreError: 37900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727517 with bad export cookie 3238072217986450132 [ 574.059317] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.060236] LustreError: 37900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 580.160896] LustreError: 40763:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 580.164369] LustreError: 40763:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 580.215952] Lustre: server umount lustre-MDT0001 complete [ 583.903734] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 584.998571] Key type lgssc unregistered [ 585.133371] LNet: 41296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.135494] LNetError: 41296:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.147315] LNet: Removed LNI 192.168.202.159@tcp [ 585.478659] Key type .llcrypt unregistered [ 585.479770] Key type ._llcrypt unregistered [ 594.105558] Key type ._llcrypt registered [ 594.106573] Key type .llcrypt registered [ 594.141419] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 594.471716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 594.530800] alg: No test for adler32 (adler32-zlib) [ 595.377417] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 595.450148] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 597.023152] Key type lgssc registered [ 597.358720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.833930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 599.841421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.985466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.235209] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 604.936853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 605.052211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 606.224503] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 607.913957] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 609.011903] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 611.168616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 611.274932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.066461] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 614.704775] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 625.638668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 635.273956] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 635.872675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 635.873661] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.875477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 635.881084] Lustre: Skipped 2 previous similar messages [ 638.458914] LustreError: 44929:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 638.483579] Lustre: server umount lustre-MDT0000 complete [ 639.672099] LustreError: 42646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727582 with bad export cookie 15004713385660626583 [ 639.674866] LustreError: 42646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 639.675031] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.388506] LustreError: 42662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.394703] LustreError: 42662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 640.395704] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 640.400663] Lustre: Skipped 2 previous similar messages [ 640.992203] LustreError: 44087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.992582] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.024150] LustreError: 42662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.024439] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 645.031336] Lustre: Skipped 1 previous similar message [ 645.821767] LustreError: 45134:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 645.824211] LustreError: 45134:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 645.870705] Lustre: server umount lustre-MDT0001 complete [ 663.726086] LustreError: 45357:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 663.728173] LustreError: 45357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 663.769139] Lustre: server umount lustre-OST0000 complete [ 669.263346] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 670.302741] Key type lgssc unregistered [ 670.436352] LNet: 46195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.438657] LNetError: 46195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.447434] LNet: Removed LNI 192.168.202.159@tcp [ 670.737884] Key type .llcrypt unregistered [ 670.738742] Key type ._llcrypt unregistered [ 679.823836] Key type ._llcrypt registered [ 679.825097] Key type .llcrypt registered [ 679.858439] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 680.161988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 680.198458] alg: No test for adler32 (adler32-zlib) [ 681.036767] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 681.112768] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 682.687181] Key type lgssc registered [ 683.050039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 685.670214] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 685.677719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 701.151379] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 706.271186] LustreError: 47538:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 732.895205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 734.707108] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 736.413644] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 898.827273] LustreError: 48245:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 898.864142] Lustre: server umount lustre-OST0000 complete [ 904.651044] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 905.813554] Key type lgssc unregistered [ 905.950447] LNet: 49083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.953809] LNetError: 49083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.962316] LNet: Removed LNI 192.168.202.159@tcp [ 906.268093] Key type .llcrypt unregistered [ 906.269198] Key type ._llcrypt unregistered [ 914.952067] Key type ._llcrypt registered [ 914.953071] Key type .llcrypt registered [ 914.990780] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 915.424521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.448346] alg: No test for adler32 (adler32-zlib) [ 916.300733] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 916.388329] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 917.975220] Key type lgssc registered [ 918.377295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 921.235765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 921.243057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.428304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.874587] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 926.953250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 927.098361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 928.599784] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 930.366607] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 931.518616] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 933.719099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 933.833062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.602540] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 937.331327] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 943.074053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 943.074230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.080189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.173182] LustreError: 52693:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 944.211834] Lustre: server umount lustre-OST0000 complete [ 947.679853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 947.683410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.687825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.551559] LustreError: 52895:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 951.555324] LustreError: 52895:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 951.593482] Lustre: server umount lustre-MDT0000 complete [ 952.886211] LustreError: 50433:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763727895 with bad export cookie 5456184151342709536 [ 952.889914] LustreError: 50433:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 952.889969] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.954570] LustreError: 53096:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 952.957320] LustreError: 53096:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 953.001233] Lustre: server umount lustre-MDT0001 complete [ 956.511435] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 957.594451] Key type lgssc unregistered [ 957.713417] LNet: 53628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.716437] LNetError: 53628:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.723326] LNet: Removed LNI 192.168.202.159@tcp [ 958.011774] Key type .llcrypt unregistered [ 958.012785] Key type ._llcrypt unregistered [ 966.443524] Key type ._llcrypt registered [ 966.445170] Key type .llcrypt registered [ 966.484733] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 966.795788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.827679] alg: No test for adler32 (adler32-zlib) [ 967.703801] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 967.795433] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 969.375152] Key type lgssc registered [ 969.752430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 972.415238] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 972.424061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 987.871243] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 992.991245] LustreError: 54972:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 1019.615151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.441528] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1023.234578] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1025.552962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.752858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.778084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1027.064208] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1030.009731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.133532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.674509] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1033.568832] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.841087] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1036.255895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1036.258046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.261992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.332991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.334810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.644118] LustreError: 57246:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1041.684765] Lustre: server umount lustre-OST0000 complete [ 1056.844171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.845299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.848534] Lustre: Skipped 1 previous similar message [ 1056.851588] Lustre: Skipped 1 previous similar message [ 1059.936552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.939411] Lustre: Skipped 2 previous similar messages [ 1063.036195] LustreError: 57625:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1063.038320] LustreError: 57625:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1063.069099] Lustre: server umount lustre-MDT0000 complete [ 1064.333768] LustreError: 54979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728007 with bad export cookie 17080801824611179598 [ 1064.339055] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1070.467967] LustreError: 57827:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1070.469570] LustreError: 57827:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1070.519625] Lustre: server umount lustre-MDT0001 complete [ 1074.100739] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1075.182965] Key type lgssc unregistered [ 1075.322525] LNet: 58360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.325935] LNetError: 58360:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.336521] LNet: Removed LNI 192.168.202.159@tcp [ 1075.674552] Key type .llcrypt unregistered [ 1075.675653] Key type ._llcrypt unregistered [ 1084.551287] Key type ._llcrypt registered [ 1084.552255] Key type .llcrypt registered [ 1084.592536] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1085.022387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.029885] alg: No test for adler32 (adler32-zlib) [ 1085.901406] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1085.996904] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1087.591217] Key type lgssc registered [ 1087.999726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1090.762171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1090.767996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.930026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.330955] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1096.246290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1096.378400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1097.734055] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1099.469580] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1100.670932] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.923357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1103.055599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.959706] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1106.897414] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1116.391207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1121.762993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1121.763022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.763832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.766813] LustreError: Skipped 1 previous similar message [ 1121.771431] Lustre: Skipped 1 previous similar message [ 1125.423036] LustreError: 61997:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1125.460575] Lustre: server umount lustre-OST0000 complete [ 1126.880411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.881105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.884072] Lustre: Skipped 2 previous similar messages [ 1132.000593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.004272] Lustre: Skipped 2 previous similar messages [ 1132.799361] LustreError: 62198:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1132.801129] LustreError: 62198:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1132.829528] Lustre: server umount lustre-MDT0000 complete [ 1134.174726] LustreError: 59713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728077 with bad export cookie 13477584917990269935 [ 1134.179700] LustreError: 59713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1134.180381] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1134.245757] LustreError: 62401:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1134.247299] LustreError: 62401:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1134.293565] Lustre: server umount lustre-MDT0001 complete [ 1137.845149] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1138.919721] Key type lgssc unregistered [ 1139.050473] LNet: 62932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.053548] LNetError: 62932:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.063491] LNet: Removed LNI 192.168.202.159@tcp [ 1139.345566] Key type .llcrypt unregistered [ 1139.346567] Key type ._llcrypt unregistered [ 1151.159825] Key type ._llcrypt registered [ 1151.161629] Key type .llcrypt registered [ 1151.197928] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1151.548455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.593108] alg: No test for adler32 (adler32-zlib) [ 1152.478639] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1152.572580] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1154.159178] Key type lgssc registered [ 1154.595572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1157.213448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1157.219058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.393707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.766041] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1162.615529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1162.746869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1164.052852] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1165.879594] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1167.052864] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1169.424631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1169.562651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.540480] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1173.388236] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1183.907526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1186.301602] Lustre: Failing over lustre-MDT0000 [ 1186.438518] LustreError: 66861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1186.469133] Lustre: server umount lustre-MDT0000 complete [ 1188.320383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1188.323170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.328109] LustreError: 64594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.332136] LustreError: 64594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1189.260201] LustreError: 66751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.346064] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.350108] Lustre: Skipped 1 previous similar message [ 1189.709771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.749776] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1189.809664] Lustre: *** cfs_fail_loc=135, val=0*** [ 1189.858241] LustreError: 67370:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1189.860634] LustreError: 67370:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -2 [ 1189.863703] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1189.863703] [ 1189.866989] LustreError: MGC192.168.202.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1189.874151] LustreError: 67358:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1189.876270] LustreError: 67358:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1189.879436] LustreError: 67358:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1189.888955] Lustre: server umount lustre-MDT0000 complete [ 1189.890880] LustreError: 67358:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1192.236109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.273372] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.386233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.417653] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1193.804641] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1194.372234] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1197.542087] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1197.550539] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1197.572342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1202.659577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.662619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1202.662981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1202.666280] Lustre: Skipped 1 previous similar message [ 1204.664465] LustreError: 68170:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1204.666327] LustreError: 68170:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1204.705455] Lustre: server umount lustre-OST0000 complete [ 1207.778406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.778637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1207.784410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.788237] Lustre: Skipped 2 previous similar messages [ 1212.221345] LustreError: 68380:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1212.224022] LustreError: 68380:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1212.251210] Lustre: server umount lustre-MDT0000 complete [ 1212.896240] LustreError: 65305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.901912] LustreError: 65305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1213.493085] LustreError: 64573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728156 with bad export cookie 9379269856685052990 [ 1213.496053] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.496163] LustreError: 64573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1213.621635] Lustre: server umount lustre-MDT0001 complete [ 1217.357366] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1218.540196] Key type lgssc unregistered [ 1218.679517] LNet: 69113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.681876] LNetError: 69113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.688484] LNet: Removed LNI 192.168.202.159@tcp [ 1219.029720] Key type .llcrypt unregistered [ 1219.031192] Key type ._llcrypt unregistered [ 1228.002376] Key type ._llcrypt registered [ 1228.004208] Key type .llcrypt registered [ 1228.055415] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1228.561196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.578643] alg: No test for adler32 (adler32-zlib) [ 1229.429962] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1229.511646] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1231.095120] Key type lgssc registered [ 1231.513993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1234.511495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1234.517478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.699315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.137509] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1240.176342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1240.314655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1241.758067] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1243.678924] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1244.900029] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1247.267940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1247.414134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.406853] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1251.293906] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1260.710755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1264.258700] Lustre: Failing over lustre-MDT0000 [ 1264.378821] LustreError: 72753:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1264.408944] Lustre: server umount lustre-MDT0000 complete [ 1266.144130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1266.146728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.149346] LustreError: 70508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.149602] Lustre: Skipped 2 previous similar messages [ 1266.157451] LustreError: 70508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1267.083969] LustreError: 70479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.655740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.694756] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.749037] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1267.806140] LustreError: 73260:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1267.808117] LustreError: 73260:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -9 [ 1267.811031] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1267.811031] [ 1267.813616] LustreError: MGC192.168.202.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1267.819869] LustreError: 73248:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1267.822756] LustreError: 73248:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1267.825082] LustreError: 73248:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1267.833054] Lustre: server umount lustre-MDT0000 complete [ 1267.834083] LustreError: 73248:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1270.067166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.106302] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1270.194727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.218310] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1271.501335] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1272.196575] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1275.366418] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1275.368539] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1275.390489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1280.481930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1280.482194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.489661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1280.491213] Lustre: Skipped 1 previous similar message [ 1282.477815] LustreError: 74064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1282.479585] LustreError: 74064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1282.517950] Lustre: server umount lustre-OST0000 complete [ 1285.601064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1285.601211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.610076] Lustre: Skipped 1 previous similar message [ 1285.613288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1285.616038] Lustre: Skipped 1 previous similar message [ 1290.720083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.721730] Lustre: Skipped 1 previous similar message [ 1295.840248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.842070] Lustre: Skipped 1 previous similar message [ 1297.887135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1297.922659] LustreError: 74269:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1297.925480] LustreError: 74269:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1297.951842] Lustre: server umount lustre-MDT0000 complete [ 1299.220057] LustreError: 72103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728242 with bad export cookie 10707478959103949267 [ 1299.222759] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.224601] LustreError: 72103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1299.342789] Lustre: server umount lustre-MDT0001 complete [ 1302.950389] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1303.946506] Key type lgssc unregistered [ 1304.070358] LNet: 75003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.073719] LNetError: 75003:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.081347] LNet: Removed LNI 192.168.202.159@tcp [ 1304.390571] Key type .llcrypt unregistered [ 1304.391722] Key type ._llcrypt unregistered [ 1308.696879] Key type ._llcrypt registered [ 1308.697899] Key type .llcrypt registered [ 1308.741615] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1308.851315] Key type .llcrypt unregistered [ 1308.852329] Key type ._llcrypt unregistered [ 1318.023254] Key type ._llcrypt registered [ 1318.024110] Key type .llcrypt registered [ 1318.063988] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1318.502725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.532908] alg: No test for adler32 (adler32-zlib) [ 1319.389588] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1319.480983] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1321.063149] Key type lgssc registered [ 1321.455094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1324.090235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1324.096552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.253053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.514043] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1329.379363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1329.502828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1330.832240] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1332.765288] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1334.102772] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1336.514120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1336.636793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.576359] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1340.429231] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1350.946583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1355.232457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1355.234807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.238792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1356.258393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1356.258712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.263232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1359.789586] LustreError: 79103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1359.825519] Lustre: server umount lustre-OST0000 complete [ 1361.376441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.377098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1361.382805] Lustre: Skipped 1 previous similar message [ 1361.386029] Lustre: Skipped 1 previous similar message [ 1366.496317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1366.499665] Lustre: Skipped 1 previous similar message [ 1367.227411] LustreError: 79305:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1367.228900] LustreError: 79305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1367.255300] Lustre: server umount lustre-MDT0000 complete [ 1368.495023] LustreError: 78456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728311 with bad export cookie 7776023435630647112 [ 1368.498158] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1368.498921] LustreError: 78456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1368.568636] LustreError: 79506:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1368.570171] LustreError: 79506:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1368.616903] Lustre: server umount lustre-MDT0001 complete [ 1372.378502] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1373.441925] Key type lgssc unregistered [ 1373.573482] LNet: 80037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.577638] LNetError: 80037:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.589483] LNet: Removed LNI 192.168.202.159@tcp [ 1373.895845] Key type .llcrypt unregistered [ 1373.896855] Key type ._llcrypt unregistered [ 1383.083170] Key type ._llcrypt registered [ 1383.084155] Key type .llcrypt registered [ 1383.123527] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1383.433533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1383.440468] alg: No test for adler32 (adler32-zlib) [ 1384.291810] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1384.372935] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1385.959155] Key type lgssc registered [ 1386.327819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1388.863629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1388.872873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.017704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.234266] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1393.933282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1394.056779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1395.324147] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1397.103179] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1398.327738] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1400.703493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1400.830589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.740533] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1404.588815] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1415.202685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1425.378576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.379140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1425.379669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1425.383804] Lustre: Skipped 1 previous similar message [ 1428.462041] LustreError: 83674:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1428.496376] Lustre: server umount lustre-OST0000 complete [ 1429.983965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.987328] LustreError: Skipped 1 previous similar message [ 1429.988452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.993123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.994980] Lustre: Skipped 1 previous similar message [ 1435.616058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.619218] Lustre: Skipped 3 previous similar messages [ 1440.736106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1440.738684] Lustre: Skipped 1 previous similar message [ 1444.319162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1444.353448] LustreError: 83876:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1444.355551] LustreError: 83876:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1444.379152] Lustre: server umount lustre-MDT0000 complete [ 1445.627144] LustreError: 82127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728388 with bad export cookie 3740734003921955432 [ 1445.630606] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1445.631569] LustreError: 82127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1445.704584] LustreError: 84078:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1445.706863] LustreError: 84078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1445.752327] Lustre: server umount lustre-MDT0001 complete [ 1449.666731] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1450.775812] Key type lgssc unregistered [ 1450.903415] LNet: 84611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.905451] LNetError: 84611:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.914330] LNet: Removed LNI 192.168.202.159@tcp [ 1451.220453] Key type .llcrypt unregistered [ 1451.221702] Key type ._llcrypt unregistered [ 1460.569081] Key type ._llcrypt registered [ 1460.570485] Key type .llcrypt registered [ 1460.617180] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1460.988877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1461.035667] alg: No test for adler32 (adler32-zlib) [ 1461.883700] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1461.965656] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1463.551170] Key type lgssc registered [ 1463.901665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1466.482934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1466.489189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.649338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.914350] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1471.835708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1471.959909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1473.315954] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1475.129958] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1476.399448] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1478.658256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1478.775544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.664356] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1482.458899] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1492.070377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1497.569816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1497.570939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.572519] LustreError: Skipped 1 previous similar message [ 1497.573435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1497.577194] Lustre: Skipped 1 previous similar message [ 1497.580697] Lustre: Skipped 1 previous similar message [ 1502.060111] LustreError: 88247:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1502.096298] Lustre: server umount lustre-OST0000 complete [ 1507.808854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1507.808927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.809672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.815443] Lustre: Skipped 1 previous similar message [ 1509.502692] LustreError: 88450:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1509.505650] LustreError: 88450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1509.532392] Lustre: server umount lustre-MDT0000 complete [ 1510.825376] LustreError: 85962:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728453 with bad export cookie 14202443590291564080 [ 1510.831160] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.904895] LustreError: 88652:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1510.907896] LustreError: 88652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1510.980771] Lustre: server umount lustre-MDT0001 complete [ 1514.900376] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1515.927689] Key type lgssc unregistered [ 1516.067389] LNet: 89183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.070847] LNetError: 89183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.079344] LNet: Removed LNI 192.168.202.159@tcp [ 1516.398252] Key type .llcrypt unregistered [ 1516.399242] Key type ._llcrypt unregistered [ 1525.997100] Key type ._llcrypt registered [ 1525.998956] Key type .llcrypt registered [ 1526.037403] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1526.459766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.471898] alg: No test for adler32 (adler32-zlib) [ 1527.334461] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1527.434575] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1529.015101] Key type lgssc registered [ 1529.443207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1532.275946] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1532.283434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1547.743270] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1552.863210] LustreError: 90526:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 1579.487230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.356305] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1583.104757] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1584.965819] LustreError: 91382:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1584.990990] Lustre: server umount lustre-OST0000 complete [ 1587.774363] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:35:30 (1763728530) [ 1590.079479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.211630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.495423] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1594.333328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1594.463974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1595.805204] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1597.571742] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1598.814270] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1601.015912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1601.125796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.027272] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1604.836766] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1616.486604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1625.056242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.059096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.064354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.106344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.106750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.110147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1627.111615] Lustre: Skipped 1 previous similar message [ 1629.870143] LustreError: 94478:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1629.872159] LustreError: 94478:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1629.911986] Lustre: server umount lustre-OST0000 complete [ 1632.225098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.231057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.232746] Lustre: Skipped 1 previous similar message [ 1637.308732] LustreError: 94680:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1637.310428] LustreError: 94680:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1637.340182] Lustre: server umount lustre-MDT0000 complete [ 1637.344838] LustreError: 92763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.350738] LustreError: 92763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1638.654482] LustreError: 93670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728581 with bad export cookie 15632843391483094365 [ 1638.657963] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1638.659640] LustreError: 93670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1638.765856] Lustre: server umount lustre-MDT0001 complete [ 1642.595434] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1643.739794] Key type lgssc unregistered [ 1643.878382] LNet: 95415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.880383] LNetError: 95415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.888400] LNet: Removed LNI 192.168.202.159@tcp [ 1644.188070] Key type .llcrypt unregistered [ 1644.189210] Key type ._llcrypt unregistered [ 1653.305256] Key type ._llcrypt registered [ 1653.306167] Key type .llcrypt registered [ 1653.349511] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1653.773541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1653.783017] alg: No test for adler32 (adler32-zlib) [ 1654.639130] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1654.717905] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1656.303197] Key type lgssc registered [ 1656.727082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1659.337809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1659.344778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.544726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.892255] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1664.671366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1664.791056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1666.132444] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1667.975771] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1669.161473] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1671.431785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1671.540791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.499404] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1675.333849] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1686.887493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1695.199865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1695.201596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.204741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1697.249131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1697.250596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1697.251121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.694692] LustreError: 99054:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1699.734726] Lustre: server umount lustre-OST0000 complete [ 1702.368737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.369715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.372339] Lustre: Skipped 1 previous similar message [ 1702.378023] Lustre: Skipped 2 previous similar messages [ 1707.135241] LustreError: 99256:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1707.139440] LustreError: 99256:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1707.168996] Lustre: server umount lustre-MDT0000 complete [ 1707.488049] LustreError: 96783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.491693] LustreError: 96783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1708.372241] LustreError: 96766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728651 with bad export cookie 7942123569414550168 [ 1708.375027] LustreError: 96766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1708.375297] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1708.441049] LustreError: 99459:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1708.442751] LustreError: 99459:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1708.488745] Lustre: server umount lustre-MDT0001 complete [ 1712.054351] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1713.073855] Key type lgssc unregistered [ 1713.205374] LNet: 99989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.207428] LNetError: 99989:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.215355] LNet: Removed LNI 192.168.202.159@tcp [ 1713.511590] Key type .llcrypt unregistered [ 1713.512615] Key type ._llcrypt unregistered [ 1721.005525] Key type ._llcrypt registered [ 1721.007358] Key type .llcrypt registered [ 1721.048855] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1721.456807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1721.507797] alg: No test for adler32 (adler32-zlib) [ 1722.392397] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1722.483198] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1724.071219] Key type lgssc registered [ 1724.532198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1727.210836] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1727.216735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1742.623319] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1747.743146] LustreError: 101188:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 1774.367189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.316958] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1778.081693] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1780.339877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.436858] LustreError: MGC192.168.202.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1780.440831] LustreError: 102100:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1780.442840] LustreError: 102100:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1780.446615] LustreError: 102100:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1780.475224] Lustre: server umount lustre-MDT0000 complete [ 1780.477079] LustreError: 102100:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1793.503725] LustreError: 102533:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1793.541191] Lustre: server umount lustre-OST0000 complete [ 1796.495227] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1799.582640] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1803.010419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1805.215792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1807.277439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1809.322775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.651152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.674115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.758960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1811.772003] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1811.817960] Lustre: lustre-MDT0000: new disk, initializing [ 1811.840687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.846735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1813.156463] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1817.340849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.360214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1817.390552] Lustre: 104865:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1817.413829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1817.415423] Lustre: Skipped 1 previous similar message [ 1817.454233] Lustre: lustre-MDT0001: new disk, initializing [ 1817.486563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1817.498745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1817.501572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1818.684901] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1821.741656] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1822.920805] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1825.135570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1825.159304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1825.257952] Lustre: lustre-OST0000: new disk, initializing [ 1825.259680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1825.287898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.076150] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1830.058788] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1831.369405] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1835.512441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1835.516207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1835.542176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1836.529161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 1837.847636] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1837.915241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1840.610887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.611298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.612128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.615635] LustreError: Skipped 1 previous similar message [ 1840.618809] Lustre: Skipped 1 previous similar message [ 1844.587072] LustreError: 107841:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1844.589204] LustreError: 107841:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1844.625965] Lustre: server umount lustre-OST0000 complete [ 1848.288067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1848.292654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.298883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.300389] Lustre: Skipped 1 previous similar message [ 1850.848081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.848722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.964323] LustreError: 108043:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1851.966423] LustreError: 108043:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1851.993213] Lustre: server umount lustre-MDT0000 complete [ 1853.176644] LustreError: 104855:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728796 with bad export cookie 2124620611487792147 [ 1853.179872] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1853.181925] LustreError: 104855:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1853.306731] Lustre: server umount lustre-MDT0001 complete [ 1856.095760] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:39:58 (1763728798) [ 1857.159617] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1862.233274] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1865.295239] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1868.591525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1870.759483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1872.688394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1874.785883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1877.087175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1877.117437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.198332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1877.209348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1877.245708] Lustre: lustre-MDT0000: new disk, initializing [ 1877.273835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.281441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1878.565451] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1882.595800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1882.618989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1882.656467] Lustre: 111369:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1882.677139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1882.679108] Lustre: Skipped 1 previous similar message [ 1882.715042] Lustre: lustre-MDT0001: new disk, initializing [ 1882.758058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1882.760873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1884.068206] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1887.085927] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1888.291061] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1890.580291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1890.607441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1890.703184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1890.729917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.731834] Lustre: Skipped 1 previous similar message [ 1892.565685] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1894.521422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1894.524039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1894.556557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1895.665401] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1897.029320] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1897.096786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1898.376308] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1898.452423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1900.002779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.003784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.004064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.006783] Lustre: Skipped 1 previous similar message [ 1900.008253] Lustre: Skipped 1 previous similar message [ 1901.103200] LustreError: 114308:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1901.106081] LustreError: 114308:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1901.147485] Lustre: server umount lustre-OST0000 complete [ 1903.583835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1903.586164] LustreError: Skipped 1 previous similar message [ 1905.120125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.120830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.121880] Lustre: Skipped 2 previous similar messages [ 1905.125020] Lustre: Skipped 1 previous similar message [ 1908.543343] LustreError: 114510:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1908.546299] LustreError: 114510:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1908.580152] Lustre: server umount lustre-MDT0000 complete [ 1909.813147] LustreError: 112151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728852 with bad export cookie 2124620611487793015 [ 1909.817164] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.817731] LustreError: 112151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1909.943350] Lustre: server umount lustre-MDT0001 complete [ 1912.773678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.980795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.295175] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1917.262241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1918.616690] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1920.441611] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1921.646634] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1923.903583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1925.836594] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1927.612837] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1940.750308] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1942.498816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1942.499171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.500796] LustreError: Skipped 1 previous similar message [ 1942.504830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1942.506445] Lustre: Skipped 1 previous similar message [ 1943.727274] LustreError: 117549:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1943.731279] LustreError: 117549:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1943.774846] Lustre: server umount lustre-OST0000 complete [ 1947.221765] Lustre: server umount lustre-MDT0000 complete [ 1947.618297] LustreError: 115183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.626319] LustreError: 115183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1948.452649] LustreError: 115168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728891 with bad export cookie 2124620611487793883 [ 1948.456976] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.457556] LustreError: 115168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1948.579552] Lustre: server umount lustre-MDT0001 complete [ 1952.325465] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1953.342917] Key type lgssc unregistered [ 1953.471392] LNet: 118484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.473314] LNetError: 118484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.480380] LNet: Removed LNI 192.168.202.159@tcp [ 1953.758990] Key type .llcrypt unregistered [ 1953.760052] Key type ._llcrypt unregistered [ 1959.616914] Key type ._llcrypt registered [ 1959.618085] Key type .llcrypt registered [ 1959.663478] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 1965.878443] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1966.181598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1966.195617] alg: No test for adler32 (adler32-zlib) [ 1967.042593] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 1967.121960] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1968.703174] Key type lgssc registered [ 1969.090983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.096839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1974.176076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1976.260899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1978.300088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1980.583831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1980.605264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1980.609771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.686714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1981.697858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1981.733473] Lustre: lustre-MDT0000: new disk, initializing [ 1981.755603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.764450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1983.228594] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1987.428981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1987.449323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1987.486286] Lustre: 121583:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1987.508295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1987.510184] Lustre: Skipped 1 previous similar message [ 1987.548432] Lustre: lustre-MDT0001: new disk, initializing [ 1987.579863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1987.590849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1987.593544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1988.867527] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1992.006045] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1993.157650] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1995.721047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1995.748734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1995.847549] Lustre: lustre-OST0000: new disk, initializing [ 1995.849457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1995.880377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.173842] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1998.915813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1998.923945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1998.965411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2001.844152] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2003.223785] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2003.293199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2004.596553] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2004.673379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2009.570259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2009.570579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.572841] LustreError: Skipped 1 previous similar message [ 2009.573427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.579018] Lustre: Skipped 1 previous similar message [ 2011.313113] LustreError: 124518:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2011.358413] Lustre: server umount lustre-OST0000 complete [ 2013.151979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2013.157219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.163844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.167884] Lustre: Skipped 1 previous similar message [ 2014.688159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.688974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.684832] LustreError: 124720:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2018.686619] LustreError: 124720:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2018.714521] Lustre: server umount lustre-MDT0000 complete [ 2019.848146] LustreError: 122362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728962 with bad export cookie 10676772498662464944 [ 2019.851027] LustreError: 122362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2019.851600] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.914491] LustreError: 124923:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2019.916604] LustreError: 124923:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2019.966957] Lustre: server umount lustre-MDT0001 complete [ 2022.378972] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:42:44 (1763728964) [ 2024.398463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.558900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.656017] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2028.599539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2028.723520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.452643] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2033.078545] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2034.643431] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2038.626198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.633976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.637880] Lustre: Skipped 2 previous similar messages [ 2041.668896] LustreError: 126934:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2041.671181] LustreError: 126934:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2041.701845] Lustre: server umount lustre-MDT0000 complete [ 2043.089324] LustreError: 126936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763728986 with bad export cookie 10676772498662465812 [ 2043.091630] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2043.092395] LustreError: 126936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2043.202276] Lustre: server umount lustre-MDT0001 complete [ 2046.426348] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:43:08 (1763728988) [ 2048.574579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2064.031409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2069.151414] LustreError: 127780:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 2094.751403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.644392] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2098.367667] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2099.039281] LustreError: 128444:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2099.041079] LustreError: 128444:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2099.080029] Lustre: server umount lustre-OST0000 complete [ 2101.681615] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:44:03 (1763729043) [ 2103.819566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.984563] LustreError: 129114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.989371] LustreError: 129114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2105.448711] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2108.545045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2109.841304] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2111.551627] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2112.827104] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2114.978086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2115.040808] LustreError: 130710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.044226] LustreError: 130710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2115.094901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.097562] Lustre: Skipped 2 previous similar messages [ 2116.717607] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2118.345752] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2123.233496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.235405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.236443] Lustre: Skipped 3 previous similar messages [ 2123.237806] Lustre: Skipped 1 previous similar message [ 2126.926651] LustreError: 131357:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2126.930630] LustreError: 131357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2126.968335] Lustre: server umount lustre-MDT0000 complete [ 2128.352044] LustreError: 129113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.356851] LustreError: 129113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2128.611767] LustreError: 129093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729071 with bad export cookie 10676772498662466463 [ 2128.617666] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.622081] LustreError: 129093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2128.730599] Lustre: server umount lustre-MDT0001 complete [ 2140.433427] Lustre: server umount lustre-OST0000 complete [ 2142.850140] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:44:45 (1763729085) [ 2144.853391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.003770] LustreError: 132433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.181677] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2148.928641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2149.050585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2149.052347] Lustre: Skipped 1 previous similar message [ 2150.258359] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2151.916045] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2153.026057] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2155.288831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2156.454043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2157.211292] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2158.924104] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2160.241566] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2160.305474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2161.549572] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2161.613292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2164.705394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2164.707352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.708289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.713849] Lustre: Skipped 1 previous similar message [ 2164.717060] Lustre: Skipped 1 previous similar message [ 2168.302240] LustreError: 135175:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2168.304637] LustreError: 135175:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2168.343539] Lustre: server umount lustre-OST0000 complete [ 2169.824524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2175.847268] Lustre: server umount lustre-MDT0000 complete [ 2177.431987] LustreError: 132457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729120 with bad export cookie 10676772498662467520 [ 2177.434944] LustreError: 132457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2177.435588] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.533546] Lustre: server umount lustre-MDT0001 complete [ 2180.707795] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:45:22 (1763729122) [ 2183.048603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2198.495495] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2203.615402] LustreError: 136223:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 2229.215361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.217555] Lustre: Skipped 1 previous similar message [ 2230.878680] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2232.587841] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2234.716070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.854090] LustreError: 136250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.896734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2236.009882] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2238.671146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2239.890666] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2241.473301] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2242.596297] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2243.820418] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2243.885708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2245.047322] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2245.111258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2249.184318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2249.187675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.192922] Lustre: Skipped 2 previous similar messages [ 2249.195293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.197883] Lustre: Skipped 4 previous similar messages [ 2251.836257] LustreError: 138998:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2251.839111] LustreError: 138998:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2251.868562] Lustre: server umount lustre-MDT0000 complete [ 2253.013297] LustreError: 137861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729196 with bad export cookie 10676772498662468528 [ 2253.017268] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2253.017697] LustreError: 137861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2253.120631] Lustre: server umount lustre-MDT0001 complete [ 2264.466658] Lustre: server umount lustre-OST0000 complete [ 2267.110609] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:46:49 (1763729209) [ 2269.242865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2284.703468] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2289.823449] LustreError: 140047:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 2317.121177] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2318.787195] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2320.862378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.010522] LustreError: 140074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.055608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2322.247670] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2324.849419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2326.077794] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2327.676519] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2328.769603] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2331.065617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2331.097358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2331.155987] Lustre: lustre-OST0001: new disk, initializing [ 2331.157555] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2333.096042] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2336.108828] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2336.248787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2336.251317] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2336.287105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2337.479728] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2337.554459] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2338.758244] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2338.823201] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2340.322067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2340.323871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.326559] Lustre: Skipped 2 previous similar messages [ 2340.328177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2340.329544] Lustre: Skipped 3 previous similar messages [ 2345.519111] LustreError: 143774:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2345.520729] LustreError: 143774:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2345.536201] Lustre: server umount lustre-OST0000 complete [ 2346.471231] LustreError: 141006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.475743] LustreError: 141006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2361.311083] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2361.361398] Lustre: server umount lustre-OST0001 complete [ 2365.919784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2365.921780] LustreError: Skipped 2 previous similar messages [ 2368.672492] Lustre: server umount lustre-MDT0000 complete [ 2369.831027] LustreError: 140054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729312 with bad export cookie 10676772498662469473 [ 2369.833975] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2369.834263] LustreError: 140054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2369.945601] Lustre: server umount lustre-MDT0001 complete [ 2371.716710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2372.943029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2374.141650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2375.318373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2377.231574] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:48:39 (1763729319) [ 2377.767566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2378.290847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2378.837276] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:48:41 (1763729321) [ 2380.918588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.006268] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2381.014830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2381.081275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.083075] Lustre: Skipped 6 previous similar messages [ 2382.265105] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2384.983923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2385.016495] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2385.025863] Lustre: 146014:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2386.360922] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2388.071366] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2389.243620] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2391.427691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2391.488270] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2393.266769] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2394.925310] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2396.140851] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2399.245305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2400.273218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2401.535411] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2401.615429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2404.321022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.322856] LustreError: Skipped 1 previous similar message [ 2408.327761] Lustre: server umount lustre-OST0000 complete [ 2421.909811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2423.730712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2424.102124] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2424.105156] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2424.105207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.465548] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2426.900150] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2426.959369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2428.151970] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2428.215061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2429.380260] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff920c05b0f000.ost_server_uuid 50 [ 2429.899611] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff920c05b0f000.ost_server_uuid in FULL state after 0 sec [ 2429.952457] LustreError: 147392:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.202.59@tcp) returned error from blocking AST (req@0000000053c864b3 x1849403805714944 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a977230a/0x942b8405026fb8a6 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.59@tcp remote: 0x5cc60d73e7ea8a49 expref: 5 pid: 146020 timeout: 2529 lvb_type: 0 [ 2429.961041] LustreError: lustre-MDT0000: A client on nid 192.168.202.59@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2429.963440] LustreError: 146010:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.59@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a977230a/0x942b8405026fb8a6 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.59@tcp remote: 0x5cc60d73e7ea8a49 expref: 6 pid: 146020 timeout: 0 lvb_type: 0 [ 2434.529908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.531640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.533021] Lustre: Skipped 8 previous similar messages [ 2434.534382] Lustre: Skipped 11 previous similar messages [ 2436.998513] Lustre: server umount lustre-OST0000 complete [ 2444.310738] Lustre: server umount lustre-MDT0000 complete [ 2444.768657] LustreError: 146025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.775931] LustreError: 146025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2445.779585] LustreError: 146007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729388 with bad export cookie 10676772498662470649 [ 2445.786875] LustreError: 146007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2445.787959] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.127366] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2460.244487] Lustre: server umount lustre-MDT0001 complete [ 2464.759640] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2465.934391] Key type lgssc unregistered [ 2466.099633] LNet: 151525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.103509] LNetError: 151525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.111602] LNet: Removed LNI 192.168.202.159@tcp [ 2466.546487] Key type .llcrypt unregistered [ 2466.548628] Key type ._llcrypt unregistered [ 2476.988735] Key type ._llcrypt registered [ 2476.989974] Key type .llcrypt registered [ 2477.034083] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2477.377358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.408385] alg: No test for adler32 (adler32-zlib) [ 2478.263894] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 2478.344055] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2479.927141] Key type lgssc registered [ 2480.328397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2483.463617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2483.474481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.678372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.945094] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2489.233889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2489.350101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2490.505534] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2492.095892] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2493.272625] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2495.350166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2495.453689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.103297] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2498.703838] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2499.561822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2507.406459] Lustre: Failing over lustre-MDT0000 [ 2507.660093] LustreError: 155145:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2507.683115] Lustre: server umount lustre-MDT0000 complete [ 2509.792690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2509.793153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.795225] LustreError: 154513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.798034] Lustre: Skipped 1 previous similar message [ 2509.801395] LustreError: 154513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2514.913112] LustreError: 152897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.917923] LustreError: 152897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2518.538332] LustreError: 152877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.59@tcp arrived at 1763729461 with bad export cookie 16035615974888610957 [ 2518.542333] LustreError: 152877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2520.032757] LustreError: 152897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.039140] LustreError: 152897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2520.813751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.854327] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2520.933734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.954261] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2522.136411] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2522.806753] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2522.810332] Lustre: lustre-MDT0000: Denying connection for new client 060b848b-aa61-4d66-ba49-2d88d4e3f7e2 (at 192.168.202.59@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2526.181366] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2530.271938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2530.275413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.280749] Lustre: Skipped 1 previous similar message [ 2530.283118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.296512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.298062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.596502] LustreError: 156077:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2535.598320] LustreError: 156077:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2535.633886] Lustre: server umount lustre-OST0000 complete [ 2536.823607] LustreError: 156278:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2536.825796] Lustre: 155644:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2536.829861] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2536.837753] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2536.842088] LustreError: 155644:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004dd391e9 x1849404341388032/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2536.846215] LustreError: 155644:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2536.848482] LustreError: 155644:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2536.850843] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2536.934273] LustreError: 156278:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2536.937030] LustreError: 156278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2536.964913] Lustre: server umount lustre-MDT0000 complete [ 2538.148299] LustreError: 152878:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729481 with bad export cookie 16035615974888611314 [ 2538.152161] LustreError: 152206:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a55b11fc x1849404341388928/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2538.153089] LustreError: 152878:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2538.153950] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2544.252561] LustreError: 156481:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2544.254313] LustreError: 156481:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2544.304173] Lustre: server umount lustre-MDT0001 complete [ 2547.662487] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2548.650138] Key type lgssc unregistered [ 2548.778450] LNet: 157013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.781514] LNetError: 157013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.791401] LNet: Removed LNI 192.168.202.159@tcp [ 2549.086413] Key type .llcrypt unregistered [ 2549.087424] Key type ._llcrypt unregistered [ 2557.542780] Key type ._llcrypt registered [ 2557.543709] Key type .llcrypt registered [ 2557.585151] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2557.921912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2557.977185] alg: No test for adler32 (adler32-zlib) [ 2558.819902] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 2558.893934] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2560.471218] Key type lgssc registered [ 2560.795726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2563.208268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2563.215552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.365706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.562667] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2568.194302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2568.320135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2569.597750] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2571.423360] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2572.602489] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2574.706888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2574.811553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.640501] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2578.324886] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2578.919403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2589.152829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2589.155204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.156330] LustreError: Skipped 1 previous similar message [ 2589.162332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2589.165370] Lustre: Skipped 1 previous similar message [ 2591.981387] LustreError: 160629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2592.014184] Lustre: server umount lustre-OST0000 complete [ 2594.273110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.273603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2594.273909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.276191] Lustre: Skipped 2 previous similar messages [ 2599.290254] LustreError: 160831:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2599.291785] LustreError: 160831:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2599.313790] Lustre: server umount lustre-MDT0000 complete [ 2599.392877] LustreError: 158383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.397160] LustreError: 158383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2600.481135] LustreError: 158364:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729543 with bad export cookie 18071157807748201537 [ 2600.484904] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.486125] LustreError: 158364:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2600.547317] LustreError: 161032:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2600.548927] LustreError: 161032:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2600.592961] Lustre: server umount lustre-MDT0001 complete [ 2604.076196] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2604.965915] Key type lgssc unregistered [ 2605.088316] LNet: 161563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.090220] LNetError: 161563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.097263] LNet: Removed LNI 192.168.202.159@tcp [ 2605.365247] Key type .llcrypt unregistered [ 2605.366286] Key type ._llcrypt unregistered [ 2616.200801] Key type ._llcrypt registered [ 2616.201679] Key type .llcrypt registered [ 2616.234248] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2616.616153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.632145] alg: No test for adler32 (adler32-zlib) [ 2617.499095] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 2617.585090] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2619.167158] Key type lgssc registered [ 2619.508057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2621.897144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2621.902796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.048132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.240942] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2626.958993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2627.087447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2628.432932] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2630.358995] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2631.524197] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2633.634942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2633.739158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.395791] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2636.960307] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2648.099274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2658.274314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.274953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.275929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.279810] Lustre: Skipped 1 previous similar message [ 2658.284699] LustreError: Skipped 1 previous similar message [ 2659.948483] LustreError: 165487:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2659.980661] Lustre: server umount lustre-OST0000 complete [ 2662.879911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2662.883148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.888639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.890208] Lustre: Skipped 1 previous similar message [ 2667.260338] LustreError: 165690:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2667.261908] LustreError: 165690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2667.285897] Lustre: server umount lustre-MDT0000 complete [ 2668.471060] LustreError: 165387:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729611 with bad export cookie 6243381238891322293 [ 2668.474018] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.476240] LustreError: 165387:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2668.536930] LustreError: 165892:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2668.538597] LustreError: 165892:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2668.594871] Lustre: server umount lustre-MDT0001 complete [ 2671.992712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2672.891313] Key type lgssc unregistered [ 2673.016372] LNet: 166422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2673.018463] LNetError: 166422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2673.028316] LNet: Removed LNI 192.168.202.159@tcp [ 2673.299031] Key type .llcrypt unregistered [ 2673.300104] Key type ._llcrypt unregistered [ 2680.956622] Key type ._llcrypt registered [ 2680.957445] Key type .llcrypt registered [ 2680.990178] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2681.336051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2681.373749] alg: No test for adler32 (adler32-zlib) [ 2682.227650] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 2682.310188] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2683.887211] Key type lgssc registered [ 2684.266819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.717747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2687.723262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.857504] Lustre: *** cfs_fail_loc=135, val=0*** [ 2688.897882] LustreError: 167909:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2688.899744] LustreError: 167909:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -2 [ 2688.902018] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2688.902018] [ 2688.904466] LustreError: MGC192.168.202.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2688.908439] LustreError: 167869:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2688.911051] LustreError: 167869:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2688.913387] LustreError: 167869:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2688.916176] LustreError: 167869:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2688.962558] Lustre: server umount lustre-MDT0000 complete [ 2688.963606] LustreError: 167869:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2691.162449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.345764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.527483] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2695.310331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2695.437054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2696.716221] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2698.433791] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2699.657533] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2700.295360] LustreError: 168312:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2700.298444] LustreError: 168312:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2700.300979] LustreError: 168312:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2700.768472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.768717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2700.769205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2700.772341] Lustre: Skipped 1 previous similar message [ 2705.888523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.891106] Lustre: Skipped 2 previous similar messages [ 2706.434852] LustreError: 169623:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2706.436593] LustreError: 169623:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2706.460678] Lustre: server umount lustre-MDT0000 complete [ 2707.648048] LustreError: 169624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729650 with bad export cookie 1487113719822991651 [ 2707.650925] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2707.652848] LustreError: 169624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2707.720489] LustreError: 169827:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2707.722271] LustreError: 169827:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2707.769602] Lustre: server umount lustre-MDT0001 complete [ 2711.135132] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2712.045337] Key type lgssc unregistered [ 2712.167384] LNet: 170359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2712.169227] LNetError: 170359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2712.176250] LNet: Removed LNI 192.168.202.159@tcp [ 2712.442610] Key type .llcrypt unregistered [ 2712.443628] Key type ._llcrypt unregistered [ 2719.646204] Key type ._llcrypt registered [ 2719.648065] Key type .llcrypt registered [ 2719.686151] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2719.788296] Key type .llcrypt unregistered [ 2719.789311] Key type ._llcrypt unregistered [ 2726.473713] Key type ._llcrypt registered [ 2726.474599] Key type .llcrypt registered [ 2726.512831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2726.866484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2726.881978] alg: No test for adler32 (adler32-zlib) [ 2727.734251] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 2727.828211] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2729.415102] Key type lgssc registered [ 2729.758250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2732.126094] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2732.131560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2747.551291] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2752.671157] LustreError: 172427:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 2779.295167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2780.992286] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2782.660225] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2784.696162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.857703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.877061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2785.989398] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2788.541661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2788.655320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2789.811299] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2791.404668] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2792.540202] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2793.434278] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2804.194612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2804.198748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.204592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.221473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.222079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2805.232751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2806.963136] LustreError: 175152:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2807.016711] Lustre: server umount lustre-OST0000 complete [ 2809.311763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2809.313595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.316675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2809.318114] Lustre: Skipped 1 previous similar message [ 2815.456054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.457503] Lustre: Skipped 3 previous similar messages [ 2820.576046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2820.578006] Lustre: Skipped 1 previous similar message [ 2823.135098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2823.167474] LustreError: 175353:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2823.170061] LustreError: 175353:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2823.192588] Lustre: server umount lustre-MDT0000 complete [ 2824.586332] LustreError: 172435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729767 with bad export cookie 8658453362603309379 [ 2824.588948] LustreError: 172435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2824.589323] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2824.655535] LustreError: 175555:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2824.658045] LustreError: 175555:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2824.707835] Lustre: server umount lustre-MDT0001 complete [ 2828.245497] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2829.156034] Key type lgssc unregistered [ 2829.270305] LNet: 176086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2829.272279] LNetError: 176086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2829.280310] LNet: Removed LNI 192.168.202.159@tcp [ 2829.553163] Key type .llcrypt unregistered [ 2829.554159] Key type ._llcrypt unregistered [ 2839.834784] Key type ._llcrypt registered [ 2839.835597] Key type .llcrypt registered [ 2839.867653] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2840.183307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2840.239201] alg: No test for adler32 (adler32-zlib) [ 2841.082202] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 2841.163045] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2842.735237] Key type lgssc registered [ 2843.138741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2845.838878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2845.844170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.025935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.297629] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2851.105249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2851.217403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.416590] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2854.256068] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2855.308112] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2857.703830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2857.820943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.639446] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2861.358278] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2872.166409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2873.239654] Lustre: Failing over lustre-MDT0000 [ 2873.304261] LustreError: 179773:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2873.328558] Lustre: server umount lustre-MDT0000 complete [ 2876.895868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2876.897522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.900398] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.903389] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2877.409141] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.409991] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.412323] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2877.414765] Lustre: Skipped 1 previous similar message [ 2882.436359] LustreError: 179611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.092546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.136617] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2886.228491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2886.257827] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2887.394417] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2887.556416] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2888.827076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2891.237477] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2891.243148] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2891.258598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2897.074053] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2906.595533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.595536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2906.596501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2906.598481] Lustre: Skipped 1 previous similar message [ 2909.358066] LustreError: 181195:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2909.359780] LustreError: 181195:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2909.398171] Lustre: server umount lustre-OST0000 complete [ 2911.712815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2911.712932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2911.713575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.713580] Lustre: Skipped 1 previous similar message [ 2911.721463] Lustre: Skipped 1 previous similar message [ 2916.604862] LustreError: 181400:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2916.607249] LustreError: 181400:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2916.630090] Lustre: server umount lustre-MDT0000 complete [ 2916.831879] LustreError: 179776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.835787] LustreError: 179776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2917.707192] LustreError: 177436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729860 with bad export cookie 2508281471765775023 [ 2917.710014] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2917.711612] LustreError: 177436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2917.819865] Lustre: server umount lustre-MDT0001 complete [ 2920.837359] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2921.707463] Key type lgssc unregistered [ 2921.819312] LNet: 182133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.821255] LNetError: 182133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2921.828263] LNet: Removed LNI 192.168.202.159@tcp [ 2922.091582] Key type .llcrypt unregistered [ 2922.092497] Key type ._llcrypt unregistered [ 2930.113575] Key type ._llcrypt registered [ 2930.114541] Key type .llcrypt registered [ 2930.152749] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2930.610248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2930.642957] alg: No test for adler32 (adler32-zlib) [ 2931.487671] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 2931.563657] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2933.143188] Key type lgssc registered [ 2933.490363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2936.235302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2936.240829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.417540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.571915] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2941.148034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2941.256134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2942.399265] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2944.127487] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2945.403645] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2947.853698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2948.026754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.218179] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2951.934735] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2962.349826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 2968.244693] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 2974.772431] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 2984.812044] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 2996.195493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.197908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2996.198754] Lustre: Skipped 1 previous similar message [ 2999.981099] LustreError: 185940:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3000.017504] Lustre: server umount lustre-OST0000 complete [ 3001.312205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.312782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3001.314995] Lustre: Skipped 1 previous similar message [ 3001.316453] Lustre: Skipped 1 previous similar message [ 3006.431995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.433514] Lustre: Skipped 1 previous similar message [ 3007.291270] LustreError: 186141:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3007.292973] LustreError: 186141:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3007.320261] Lustre: server umount lustre-MDT0000 complete [ 3008.414136] LustreError: 183483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763729951 with bad export cookie 3635555871187593873 [ 3008.416628] LustreError: 183483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3008.417021] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3008.482306] LustreError: 186342:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3008.484065] LustreError: 186342:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3008.527891] Lustre: server umount lustre-MDT0001 complete [ 3011.858657] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3012.701550] Key type lgssc unregistered [ 3012.811344] LNet: 186873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3012.813284] LNetError: 186873:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3012.820278] LNet: Removed LNI 192.168.202.159@tcp [ 3013.076732] Key type .llcrypt unregistered [ 3013.077684] Key type ._llcrypt unregistered [ 3021.758575] Key type ._llcrypt registered [ 3021.759455] Key type .llcrypt registered [ 3021.796472] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3022.115513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3022.156593] alg: No test for adler32 (adler32-zlib) [ 3023.008648] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 3023.095358] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3024.671118] Key type lgssc registered [ 3025.008370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3027.362058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3027.367256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.536658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.775949] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3032.485504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3032.606547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3033.766266] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3035.362947] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3036.445859] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3038.571584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3038.684527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3040.372346] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3042.103965] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3042.791926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3049.485516] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3055.665890] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3063.320977] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3068.212903] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3078.625424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3078.627569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.628762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.630717] Lustre: Skipped 1 previous similar message [ 3083.052460] LustreError: 192267:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3083.082277] Lustre: server umount lustre-OST0000 complete [ 3088.865182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.865386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3088.865735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3088.865739] Lustre: Skipped 1 previous similar message [ 3088.868053] Lustre: Skipped 1 previous similar message [ 3088.871378] LustreError: Skipped 1 previous similar message [ 3090.303549] LustreError: 192468:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3090.305248] LustreError: 192468:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3090.330253] Lustre: server umount lustre-MDT0000 complete [ 3091.513144] LustreError: 188225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763730034 with bad export cookie 11438481648609392358 [ 3091.516441] LustreError: 188225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3091.516799] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.584309] LustreError: 192669:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3091.587234] LustreError: 192669:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3091.642297] Lustre: server umount lustre-MDT0001 complete [ 3094.857507] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3095.862256] Key type lgssc unregistered [ 3095.993490] LNet: 193199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3095.995688] LNetError: 193199:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3096.006427] LNet: Removed LNI 192.168.202.159@tcp [ 3096.291212] Key type .llcrypt unregistered [ 3096.292239] Key type ._llcrypt unregistered [ 3104.995972] Key type ._llcrypt registered [ 3104.997018] Key type .llcrypt registered [ 3105.033126] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3105.488307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3105.528516] alg: No test for adler32 (adler32-zlib) [ 3106.385551] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 3106.469516] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3108.055119] Key type lgssc registered [ 3108.398707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3111.078355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3111.083981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.248487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.613553] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3116.396693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3116.522040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3117.761666] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3119.525437] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3120.920566] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3123.812194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3124.029611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.463659] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3129.078679] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3137.388605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3146.720516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3146.724684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.731967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3147.748296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.748325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3147.748455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.199148] LustreError: 197002:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3150.257390] Lustre: server umount lustre-OST0000 complete [ 3152.352235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3152.355869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.362264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.364922] Lustre: Skipped 1 previous similar message [ 3157.985738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3157.989733] Lustre: Skipped 2 previous similar messages [ 3158.094217] LustreError: 197204:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3158.098463] LustreError: 197204:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3158.133909] Lustre: server umount lustre-MDT0000 complete [ 3159.889928] LustreError: 194593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763730102 with bad export cookie 18002810810983519448 [ 3159.895820] LustreError: 194593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3159.896954] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3159.988699] LustreError: 197408:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3159.992061] LustreError: 197408:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3160.049218] Lustre: server umount lustre-MDT0001 complete [ 3163.830285] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3164.859366] Key type lgssc unregistered [ 3164.977366] LNet: 197939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3164.979235] LNetError: 197939:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3164.985249] LNet: Removed LNI 192.168.202.159@tcp [ 3165.255891] Key type .llcrypt unregistered [ 3165.256820] Key type ._llcrypt unregistered [ 3172.153400] Key type ._llcrypt registered [ 3172.154294] Key type .llcrypt registered [ 3172.186150] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3172.498342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3172.533388] alg: No test for adler32 (adler32-zlib) [ 3173.391518] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 3173.471240] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3175.047106] Key type lgssc registered [ 3175.370719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3177.735332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3177.740775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.892354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.040270] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3182.695144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3182.806184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3183.938973] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3185.633381] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3186.656405] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3188.650249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3188.757704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.797960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3190.377419] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3191.988169] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3197.921670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3197.922119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.924212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3197.926179] Lustre: Skipped 1 previous similar message [ 3201.965103] LustreError: 201557:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3201.995460] Lustre: server umount lustre-OST0000 complete [ 3203.551822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3203.554073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.558153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3203.559597] Lustre: Skipped 1 previous similar message [ 3208.160207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3208.160750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.274258] LustreError: 201759:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3209.275768] LustreError: 201759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3209.300710] Lustre: server umount lustre-MDT0000 complete [ 3210.353113] LustreError: 199144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763730153 with bad export cookie 16203025983884749802 [ 3210.355899] LustreError: 199144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3210.356501] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.416308] LustreError: 201962:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3210.417900] LustreError: 201962:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3210.463048] Lustre: server umount lustre-MDT0001 complete [ 3213.528509] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3214.410483] Key type lgssc unregistered [ 3214.520326] LNet: 202493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3214.522362] LNetError: 202493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3214.529296] LNet: Removed LNI 192.168.202.159@tcp [ 3214.781968] Key type .llcrypt unregistered [ 3214.782814] Key type ._llcrypt unregistered [ 3221.214962] Key type ._llcrypt registered [ 3221.215967] Key type .llcrypt registered [ 3221.246753] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3221.516495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3221.524069] alg: No test for adler32 (adler32-zlib) [ 3222.360486] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 3222.437319] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3224.007106] Key type lgssc registered [ 3224.301966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3226.565526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3226.571353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.709338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.788327] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3231.318384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3231.429648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3232.606608] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3234.552731] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3235.980173] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3238.381053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3238.491712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.211333] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3241.878905] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3252.834678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3260.061046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3260.081172] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3260.145609] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3261.768455] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3263.341943] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3269.103681] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 3279.852604] Lustre: Permanently deactivating lustre-OST0001 [ 3279.862149] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3287.538644] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3292.342285] Lustre: Permanently reactivating lustre-OST0001 [ 3292.351908] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3292.355608] Lustre: Skipped 2 previous similar messages [ 3294.644892] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.649923] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3294.652287] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3294.659781] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3294.678144] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3304.935887] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.936546] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3304.937644] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3304.945587] Lustre: Skipped 2 previous similar messages [ 3304.949769] LustreError: Skipped 1 previous similar message [ 3305.979087] LustreError: 207199:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3306.002936] Lustre: server umount lustre-OST0001 complete [ 3308.513736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3308.517911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.525308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.528642] Lustre: Skipped 1 previous similar message [ 3309.536373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3309.541423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3310.053333] LustreError: 206838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.061226] LustreError: 206838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3315.171924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3315.174526] LustreError: 205884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.175792] Lustre: Skipped 3 previous similar messages [ 3315.185455] LustreError: 205884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3320.293094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3320.295946] LustreError: 205314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.296906] Lustre: Skipped 1 previous similar message [ 3320.305139] LustreError: 205314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3322.335229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3322.359170] LustreError: 207400:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3322.362694] LustreError: 207400:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3322.416767] Lustre: server umount lustre-OST0000 complete [ 3325.410949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.417626] Lustre: Skipped 1 previous similar message [ 3330.320630] LustreError: 207602:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3330.323929] LustreError: 207602:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3330.361289] Lustre: server umount lustre-MDT0000 complete [ 3330.528063] LustreError: 203716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.532085] LustreError: 203716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3332.071605] LustreError: 203695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763730275 with bad export cookie 7534529304879668880 [ 3332.077315] LustreError: 203695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3332.077667] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3332.231178] Lustre: server umount lustre-MDT0001 complete [ 3336.190619] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3337.389810] Key type lgssc unregistered [ 3337.538488] LNet: 208336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3337.543071] LNetError: 208336:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3337.552424] LNet: Removed LNI 192.168.202.159@tcp [ 3337.989594] Key type .llcrypt unregistered [ 3337.991572] Key type ._llcrypt unregistered [ 3352.291195] Key type ._llcrypt registered [ 3352.293194] Key type .llcrypt registered [ 3352.330262] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3358.498262] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3358.901438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3358.907472] alg: No test for adler32 (adler32-zlib) [ 3359.778226] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 3359.856211] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3361.431125] Key type lgssc registered [ 3361.871692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3365.511186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3368.322042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3370.764134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3373.463678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3376.584316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3376.620785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3376.630511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.741173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3377.756100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3377.798306] Lustre: lustre-MDT0000: new disk, initializing [ 3377.834988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.847221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3379.600700] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3385.221948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3385.263639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3385.317699] Lustre: 212257:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3385.359499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3385.362682] Lustre: Skipped 1 previous similar message [ 3385.408130] Lustre: lustre-MDT0001: new disk, initializing [ 3385.464663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3385.489856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3385.495744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3387.044480] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3390.839132] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3392.319053] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3394.986917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3395.017568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3395.155751] Lustre: lustre-OST0000: new disk, initializing [ 3395.158975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3395.198860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.478367] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3400.197350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3400.204925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3400.251369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3401.252992] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3402.627066] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3402.698323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3404.289332] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3404.383896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3405.283994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.284157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3405.285701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3405.287071] Lustre: Skipped 1 previous similar message [ 3405.289487] LustreError: Skipped 1 previous similar message [ 3407.150088] LustreError: 215194:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3407.183811] Lustre: server umount lustre-OST0000 complete [ 3410.403630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.405606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.410381] Lustre: Skipped 1 previous similar message [ 3410.417720] Lustre: Skipped 2 previous similar messages [ 3415.520015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3415.521562] Lustre: Skipped 1 previous similar message [ 3420.640916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3420.644450] Lustre: Skipped 1 previous similar message [ 3423.199211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3423.243480] LustreError: 215395:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3423.246215] LustreError: 215395:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3423.279940] Lustre: server umount lustre-MDT0000 complete [ 3424.915226] LustreError: 213036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763730367 with bad export cookie 12672826734766005131 [ 3424.921537] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.921826] LustreError: 213036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3425.026218] LustreError: 215598:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3425.029175] LustreError: 215598:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3425.089774] Lustre: server umount lustre-MDT0001 complete [ 3429.035395] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 08:06:11 (1763730371) [ 3431.919902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.102709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.502505] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3436.882920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3437.067852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3438.706386] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3441.167531] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3442.871558] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3445.159332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3445.346599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.785326] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3449.698590] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3460.529452] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3466.081834] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3475.697454] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3485.288246] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3499.286775] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3499.289119] Lustre: Skipped 1 previous similar message [ 3517.787142] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3517.789082] Lustre: Skipped 1 previous similar message [ 3556.313143] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3556.315543] Lustre: Skipped 4 previous similar messages [ 3620.398427] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3620.404120] Lustre: Skipped 7 previous similar messages [ 3633.119734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3633.121524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3633.124566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3636.194621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.381096] LustreError: 219769:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3638.382802] LustreError: 219769:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3638.411521] Lustre: server umount lustre-OST0000 complete [ 3641.312841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.313415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3641.315992] Lustre: Skipped 1 previous similar message [ 3641.317505] Lustre: Skipped 2 previous similar messages [ 3654.111143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3654.141614] LustreError: 219971:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3654.143370] LustreError: 219971:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3654.166891] Lustre: server umount lustre-MDT0000 complete [ 3655.537446] LustreError: 218489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763730598 with bad export cookie 12672826734766006013 [ 3655.540025] LustreError: 218489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3655.540666] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3655.651404] Lustre: server umount lustre-MDT0001 complete [ 3659.766953] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3661.235403] Key type lgssc unregistered [ 3661.389634] LNet: 220709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3661.393958] LNetError: 220709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3661.402608] LNet: Removed LNI 192.168.202.159@tcp [ 3661.849365] Key type .llcrypt unregistered [ 3661.851801] Key type ._llcrypt unregistered [ 3671.763122] Key type ._llcrypt registered [ 3671.764209] Key type .llcrypt registered [ 3671.796921] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3672.259734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3672.266810] alg: No test for adler32 (adler32-zlib) [ 3673.137223] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 3673.231566] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3674.831288] Key type lgssc registered [ 3675.311488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3678.390189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3678.400091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3679.624480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.149974] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3685.153189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3685.368993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3686.989615] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3689.329094] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3690.978244] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3694.081389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3694.236419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3696.625195] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3698.938450] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3715.046363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.047072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3715.048964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3715.052775] Lustre: Skipped 1 previous similar message [ 3720.164923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3720.168210] Lustre: Skipped 2 previous similar messages [ 3720.562142] LustreError: 224499:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3720.613673] Lustre: server umount lustre-OST0000 complete [ 3725.282517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.283695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.289271] Lustre: Skipped 1 previous similar message [ 3725.303633] Lustre: Skipped 1 previous similar message [ 3728.390500] LustreError: 224705:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3728.394016] LustreError: 224705:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3728.426625] Lustre: server umount lustre-MDT0000 complete [ 3729.739219] LustreError: 224351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763730672 with bad export cookie 12796233526037660970 [ 3729.742392] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.747921] LustreError: 224351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3729.845913] LustreError: 224907:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3729.847904] LustreError: 224907:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3729.905571] Lustre: server umount lustre-MDT0001 complete [ 3734.022111] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3735.391235] Key type lgssc unregistered [ 3735.537519] LNet: 225437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3735.542561] LNetError: 225437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3735.551586] LNet: Removed LNI 192.168.202.159@tcp [ 3736.039252] Key type .llcrypt unregistered [ 3736.042676] Key type ._llcrypt unregistered [ 3744.873508] Key type ._llcrypt registered [ 3744.875144] Key type .llcrypt registered [ 3744.932887] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3745.091128] Key type .llcrypt unregistered [ 3745.093290] Key type ._llcrypt unregistered [ 3755.571752] Key type ._llcrypt registered [ 3755.573039] Key type .llcrypt registered [ 3755.628569] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3763.541282] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3763.851276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3763.889366] alg: No test for adler32 (adler32-zlib) [ 3764.759224] Lustre: Lustre: Build Version: 2.16.61_40_g9cf2b75 [ 3764.844450] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3766.423162] Key type lgssc registered [ 3766.942769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3771.382419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3774.237640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3777.465157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3780.660306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3781.225409] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3645 sec ========== 08:12:03 (1763730723) [ 3781.872220] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:12:04 (1763730724) === [ 3783.343909] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:12:05 (1763730725) === [ 3802.334953] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3803.802880] Key type lgssc unregistered [ 3803.981734] LNet: 231519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3803.985974] LNetError: 231519:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3803.994736] LNet: Removed LNI 192.168.202.159@tcp [ 3804.445309] Key type .llcrypt unregistered [ 3804.447064] Key type ._llcrypt unregistered