[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 533831511 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002014] Switched APIC routing to physical x2apic. [ 0.003034] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008022] pid_max: default: 32768 minimum: 301 [ 0.010068] LSM: Security Framework initializing [ 0.011124] Yama: becoming mindful. [ 0.013074] SELinux: Initializing. [ 0.014167] *** VALIDATE selinux *** [ 0.024337] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029757] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030204] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031148] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033010] *** VALIDATE tmpfs *** [ 0.034424] *** VALIDATE proc *** [ 0.036178] *** VALIDATE cgroup *** [ 0.037010] *** VALIDATE cgroup2 *** [ 0.038309] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041000] Spectre V2 : User space: Vulnerable [ 0.041020] Speculative Store Bypass: Vulnerable [ 0.043885] debug: unmapping init [mem 0xffffffff84259000-0xffffffff84260fff] [ 0.045366] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046904] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047047] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049022] ... generic registers: 4 [ 0.050024] ... value mask: 0000ffffffffffff [ 0.051027] ... max period: 00007fffffffffff [ 0.052019] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054408] rcu: Hierarchical SRCU implementation. [ 0.056834] smp: Bringing up secondary CPUs ... [ 0.057692] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.068245] smp: Brought up 1 node, 4 CPUs [ 0.070031] smpboot: Max logical packages: 1 [ 0.071025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.261052] node 0 deferred pages initialised in 187ms [ 0.265237] devtmpfs: initialized [ 0.266364] x86/mm: Memory block size: 128MB [ 0.269385] gcov: version magic: 0x41383552 [ 0.272401] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.273139] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.275540] pinctrl core: initialized pinctrl subsystem [ 0.277265] [ 0.277873] ************************************************************* [ 0.281029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283019] ** ** [ 0.285016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.287018] ** ** [ 0.290022] ** This means that this kernel is built to expose internal ** [ 0.292022] ** IOMMU data structures, which may compromise security on ** [ 0.294027] ** your system. ** [ 0.297021] ** ** [ 0.299021] ** If you see this message and you are not debugging the ** [ 0.301023] ** kernel, report this immediately to your vendor! ** [ 0.303023] ** ** [ 0.306022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.308016] ************************************************************* [ 0.311917] NET: Registered protocol family 16 [ 0.313488] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.316099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.319121] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.322150] cpuidle: using governor menu [ 0.323925] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.326724] PCI: Using configuration type 1 for base access [ 0.330141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.347112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.350052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.353145] cryptd: max_cpu_qlen set to 1000 [ 0.355071] ACPI: Added _OSI(Module Device) [ 0.356028] ACPI: Added _OSI(Processor Device) [ 0.358016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.359018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.363334] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.369559] ACPI: Interpreter enabled [ 0.371057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.372023] ACPI: Using IOAPIC for interrupt routing [ 0.374111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.377443] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.386000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.387032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.389019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.393082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.394440] acpiphp: Slot [2] registered [ 0.396198] acpiphp: Slot [5] registered [ 0.398151] acpiphp: Slot [6] registered [ 0.399122] acpiphp: Slot [7] registered [ 0.401181] acpiphp: Slot [8] registered [ 0.402099] acpiphp: Slot [9] registered [ 0.403109] acpiphp: Slot [10] registered [ 0.405132] acpiphp: Slot [3] registered [ 0.406089] acpiphp: Slot [4] registered [ 0.407139] acpiphp: Slot [11] registered [ 0.408074] acpiphp: Slot [12] registered [ 0.409075] acpiphp: Slot [13] registered [ 0.413226] acpiphp: Slot [14] registered [ 0.414148] acpiphp: Slot [15] registered [ 0.415000] acpiphp: Slot [16] registered [ 0.416075] acpiphp: Slot [17] registered [ 0.417219] acpiphp: Slot [18] registered [ 0.419113] acpiphp: Slot [19] registered [ 0.420146] acpiphp: Slot [20] registered [ 0.422186] acpiphp: Slot [21] registered [ 0.423184] acpiphp: Slot [22] registered [ 0.425177] acpiphp: Slot [23] registered [ 0.426128] acpiphp: Slot [24] registered [ 0.427137] acpiphp: Slot [25] registered [ 0.429113] acpiphp: Slot [26] registered [ 0.430145] acpiphp: Slot [27] registered [ 0.432139] acpiphp: Slot [28] registered [ 0.433132] acpiphp: Slot [29] registered [ 0.434239] acpiphp: Slot [30] registered [ 0.436334] acpiphp: Slot [31] registered [ 0.438129] PCI host bridge to bus 0000:00 [ 0.439032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.442043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.444041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.447039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.451033] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.454031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.456236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.460240] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.464369] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.474894] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.480000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.481040] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.483037] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.486047] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.488542] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.492273] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.495084] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.498880] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.503021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.515000] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.520000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.525000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.529000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.533022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.548027] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.557581] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.564020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.570025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.588023] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.599861] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.606025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.609039] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.625025] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.633329] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.638016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.643021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.658033] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.670998] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.677026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.684028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.693031] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.707417] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.710000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.714036] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.726034] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.739862] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.742629] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.745327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.747357] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.749269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.755137] iommu: Default domain type: Passthrough [ 0.757698] SCSI subsystem initialized [ 0.758171] ACPI: bus type USB registered [ 0.760143] usbcore: registered new interface driver usbfs [ 0.761135] usbcore: registered new interface driver hub [ 0.763157] usbcore: registered new device driver usb [ 0.765234] pps_core: LinuxPPS API ver. 1 registered [ 0.767015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.769101] PTP clock support registered [ 0.772227] EDAC MC: Ver: 3.0.0 [ 0.774155] PCI: Using ACPI for IRQ routing [ 0.776000] NetLabel: Initializing [ 0.777016] NetLabel: domain hash size = 128 [ 0.778011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.779100] NetLabel: unlabeled traffic allowed by default [ 0.782217] vgaarb: loaded [ 0.783264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.786020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.791659] clocksource: Switched to clocksource kvm-clock [ 0.917275] VFS: Disk quotas dquot_6.6.0 [ 0.919096] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.922025] *** VALIDATE ramfs *** [ 0.923386] *** VALIDATE hugetlbfs *** [ 0.925211] pnp: PnP ACPI init [ 0.928928] pnp: PnP ACPI: found 6 devices [ 0.948545] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.952329] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.954718] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.957169] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.959939] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.962763] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.966298] NET: Registered protocol family 2 [ 0.969892] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.974572] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.979419] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.985378] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.988837] TCP: Hash tables configured (established 65536 bind 65536) [ 0.991871] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.995036] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.997927] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.001582] NET: Registered protocol family 1 [ 1.005878] RPC: Registered named UNIX socket transport module. [ 1.008400] RPC: Registered udp transport module. [ 1.010300] RPC: Registered tcp transport module. [ 1.012625] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.014867] NET: Registered protocol family 44 [ 1.016363] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.018715] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.021399] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.024529] PCI: CLS 0 bytes, default 64 [ 1.026570] Unpacking initramfs... [ 2.589502] debug: unmapping init [mem 0xffff9e0fbcc54000-0xffff9e0fbffbffff] [ 2.593427] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.596547] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.599706] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.148733] Initialise system trusted keyrings [ 3.150033] Key type blacklist registered [ 3.151706] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.162373] zbud: loaded [ 3.165648] *** VALIDATE nfs *** [ 3.166600] *** VALIDATE nfs4 *** [ 3.168338] pstore: using deflate compression [ 3.171806] Platform Keyring initialized [ 3.313280] NET: Registered protocol family 38 [ 3.315833] Key type asymmetric registered [ 3.317363] Asymmetric key parser 'x509' registered [ 3.320860] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.324493] io scheduler mq-deadline registered [ 3.326440] io scheduler kyber registered [ 3.328903] io scheduler bfq registered [ 3.331718] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.334438] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.336915] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.340635] ACPI: Power Button [PWRF] [ 3.437928] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.529277] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.713679] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.804739] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.005350] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.034905] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.065367] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.071937] Non-volatile memory driver v1.3 [ 4.073903] Linux agpgart interface v0.103 [ 4.109928] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 4.114759] vda: detected capacity change from 0 to 68526080 [ 4.131386] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.134203] vdb: detected capacity change from 0 to 1073741824 [ 4.155285] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.158823] vdc: detected capacity change from 0 to 2621440000 [ 4.183082] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.186625] vdd: detected capacity change from 0 to 2621440000 [ 4.206807] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.210288] vde: detected capacity change from 0 to 4294967296 [ 4.232782] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.236511] vdf: detected capacity change from 0 to 4294967296 [ 4.246731] libphy: Fixed MDIO Bus: probed [ 4.259620] usbcore: registered new interface driver usbserial_generic [ 4.262345] usbserial: USB Serial support registered for generic [ 4.265067] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.270481] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.273173] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.276643] mousedev: PS/2 mouse device common for all mice [ 4.281404] rtc_cmos 00:05: RTC can wake from S4 [ 4.283622] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.288984] rtc_cmos 00:05: registered as rtc0 [ 4.297881] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.300566] intel_pstate: CPU model not supported [ 4.307416] hid: raw HID events driver (C) Jiri Kosina [ 4.308225] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.313309] usbcore: registered new interface driver usbhid [ 4.318319] usbhid: USB HID core driver [ 4.320306] drop_monitor: Initializing network drop monitor service [ 4.323278] Initializing XFRM netlink socket [ 4.323525] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.325407] NET: Registered protocol family 10 [ 4.331183] Segment Routing with IPv6 [ 4.332740] NET: Registered protocol family 17 [ 4.335053] mpls_gso: MPLS GSO support [ 4.343596] RAS: Correctable Errors collector initialized. [ 4.346027] AVX version of gcm_enc/dec engaged. [ 4.347492] AES CTR mode by8 optimization enabled [ 4.440072] sched_clock: Marking stable (4440032297, 0)->(5514718389, -1074686092) [ 4.443947] registered taskstats version 1 [ 4.445900] Loading compiled-in X.509 certificates [ 4.447754] zswap: loaded using pool lzo/zbud [ 4.473694] Key type big_key registered [ 4.486315] Key type encrypted registered [ 4.488324] ima: No TPM chip found, activating TPM-bypass! [ 4.490217] ima: Allocated hash algorithm: sha1 [ 4.491813] ima: No architecture policies found [ 4.495336] evm: Initialising EVM extended attributes: [ 4.497279] evm: security.selinux [ 4.498446] evm: security.ima [ 4.499393] evm: security.capability [ 4.500825] evm: HMAC attrs: 0x1 [ 4.503252] rtc_cmos 00:05: setting system clock to 2025-11-17 03:47:41 UTC (1763351261) [ 4.509820] debug: unmapping init [mem 0xffffffff85203000-0xffffffff853fffff] [ 4.512306] debug: unmapping init [mem 0xffffffff83f82000-0xffffffff84258fff] [ 4.521100] Write protecting the kernel read-only data: 28672k [ 4.524467] debug: unmapping init [mem 0xffffffff82603000-0xffffffff827fffff] [ 4.527046] debug: unmapping init [mem 0xffffffff82f14000-0xffffffff82ffffff] [ 4.574202] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.583073] systemd[1]: Detected virtualization kvm. [ 4.585310] systemd[1]: Detected architecture x86-64. [ 4.587427] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.616246] systemd[1]: No hostname configured. [ 4.617963] systemd[1]: Set hostname to . [ 4.620196] random: systemd: uninitialized urandom read (16 bytes read) [ 4.622969] systemd[1]: Initializing machine ID from random generator. [ 4.655298] random: ln: uninitialized urandom read (6 bytes read) [ 4.785639] random: systemd: uninitialized urandom read (16 bytes read) [ 4.788945] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.793405] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.798343] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.488098] device-mapper: uevent: version 1.0.3 [ 5.490292] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.268392] virtio_net virtio0 ens2: renamed from eth0 [ 6.304226] random: fast init done [ 6.304741] scsi host0: ata_piix [ 6.351660] scsi host1: ata_piix [ 6.354383] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.356866] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.191636] dracut-initqueue[579]: RTNETLINK answers: File exists [ 11.012194] random: crng init done [ 11.013341] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.580271] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.860948] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.113323] SELinux: Disabled at runtime. [ 13.179951] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.190632] systemd[1]: Detected virtualization kvm. [ 13.192831] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.735239] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.740912] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.748672] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.755269] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.760400] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.767431] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.771722] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ 13.906215] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.194497] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.500329] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.512188] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.635936] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.654796] EDAC sbridge: Ver: 1.1.2 [ 16.400703] Key type dns_resolver registered [ 16.726073] NFS: Registering the id_resolver key type [ 16.727656] Key type id_resolver registered [ 16.729073] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg130-server login: [ 67.531590] libcfs: loading out-of-tree module taints kernel. [ 67.552397] Key type ._llcrypt registered [ 67.554054] Key type .llcrypt registered [ 67.680710] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 82.550796] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 84.019902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.044859] alg: No test for adler32 (adler32-zlib) [ 85.504504] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 86.117038] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 87.815148] Key type lgssc registered [ 89.096175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.510024] hrtimer: interrupt took 7000996 ns [ 103.900941] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 105.227550] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.221447] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.532660] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 126.055165] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.303341] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 154.316151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.370391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.385710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.680179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.716538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.785748] Lustre: lustre-MDT0000: new disk, initializing [ 155.901097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.917972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.069308] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.370361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.449152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 172.535094] Lustre: 6508:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.565267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.582239] Lustre: Skipped 1 previous similar message [ 172.654829] Lustre: lustre-MDT0001: new disk, initializing [ 172.721794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.750502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.777099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.379590] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.201237] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.151220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.223337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 190.491567] Lustre: lustre-OST0000: new disk, initializing [ 190.494090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.550718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.185933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.196080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.265660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.852177] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.279572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.378559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 209.563436] Lustre: lustre-OST0001: new disk, initializing [ 209.579670] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.671060] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.064542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.086504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 215.172064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.418545] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.147771] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.538592] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.724258] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.215976] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing yml_node [ 249.053614] Lustre: DEBUG MARKER: Client: 2.16.61.41 [ 251.018989] Lustre: DEBUG MARKER: MDS: 2.16.61.41 [ 253.478767] Lustre: DEBUG MARKER: OSS: 2.16.61.41 [ 255.130411] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 22:51:50 EST 2025 [ 272.034581] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 273.622756] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.784936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 290.788526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.811025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.809469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.812343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.858671] Lustre: Skipped 1 previous similar message [ 294.408053] LustreError: 12217:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 294.508991] Lustre: server umount lustre-MDT0000 complete [ 298.311485] LustreError: 6501:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351555 with bad export cookie 9864446870982014782 [ 298.316672] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.325224] LustreError: 6501:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 302.050952] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.050988] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.064089] Lustre: Skipped 1 previous similar message [ 302.085253] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 302.101647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.107592] Lustre: Skipped 3 previous similar messages [ 303.071917] LustreError: 10288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.081805] LustreError: 10288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 304.495103] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 304.497889] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 304.617650] Lustre: server umount lustre-MDT0001 complete [ 308.264087] LustreError: 12619:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 308.266979] LustreError: 12619:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 308.361034] Lustre: server umount lustre-OST0000 complete [ 312.225357] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 312.228088] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 312.343935] Lustre: server umount lustre-OST0001 complete [ 317.823896] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 325.941143] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 335.608944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 342.453344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.274084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 355.632693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 362.084586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 362.147825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.332330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 362.357744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 362.431404] Lustre: lustre-MDT0000: new disk, initializing [ 362.519123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.540771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 366.095520] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 375.200139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 375.254610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 375.331702] Lustre: 15007:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 375.395343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 375.398795] Lustre: Skipped 1 previous similar message [ 375.480433] Lustre: lustre-MDT0001: new disk, initializing [ 375.597582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 375.625425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 375.630909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 378.842839] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 385.892936] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 388.813745] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 394.247210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 394.328525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 394.608447] Lustre: lustre-OST0000: new disk, initializing [ 394.618177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 394.697933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.939957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 395.953401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 396.095698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 400.006361] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 406.779332] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 409.887324] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 410.032169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 413.827730] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 414.022925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 416.740878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 416.742284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 416.753152] Lustre: Skipped 1 previous similar message [ 416.755859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 416.768699] Lustre: Skipped 3 previous similar messages [ 421.861989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 421.866373] Lustre: Skipped 1 previous similar message [ 421.967910] LustreError: 17944:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 421.974812] LustreError: 17944:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 422.066362] Lustre: server umount lustre-OST0000 complete [ 426.979167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.981362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 426.987772] Lustre: Skipped 2 previous similar messages [ 431.472756] LustreError: 18146:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 431.482801] LustreError: 18146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 431.588123] Lustre: server umount lustre-MDT0000 complete [ 432.098528] LustreError: 15017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.108147] LustreError: 15017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 434.239040] LustreError: 14998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351691 with bad export cookie 9864446870982016315 [ 434.250493] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 434.489751] Lustre: server umount lustre-MDT0001 complete [ 439.435421] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:54:55 (1763351695) [ 454.571637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.920967] LustreError: 19482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.989203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.296291] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 460.261769] LustreError: 19482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.385858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 468.831592] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 473.646399] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 476.679765] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 478.508800] Lustre: Failing over lustre-MDT0000 [ 478.628397] LustreError: 20824:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 478.634146] LustreError: 20824:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 478.752941] Lustre: server umount lustre-MDT0000 complete [ 480.229343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.243322] Lustre: Skipped 1 previous similar message [ 480.244355] LustreError: 19482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.256211] LustreError: 19482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 495.706460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.785278] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.875868] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 501.221279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 501.222254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 501.272482] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 505.070376] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 510.246195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.263802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.266414] Lustre: Skipped 3 previous similar messages [ 514.359506] LustreError: 21802:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 514.372936] LustreError: 21802:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 514.450892] Lustre: server umount lustre-MDT0000 complete [ 515.554528] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.578688] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 517.691514] LustreError: 19463:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351774 with bad export cookie 9864446870982017729 [ 517.696586] LustreError: 19463:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 517.700740] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.035788] Lustre: server umount lustre-MDT0001 complete [ 525.329680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.819213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.824460] Lustre: Skipped 2 previous similar messages [ 529.179485] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 536.309514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 539.632531] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 544.336187] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 547.576715] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 549.314311] Lustre: Failing over lustre-MDT0000 [ 549.542039] Lustre: server umount lustre-MDT0000 complete [ 551.904156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.905753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 551.906190] LustreError: 22481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.906232] Lustre: Skipped 2 previous similar messages [ 551.935361] LustreError: 22481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 565.454817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.531855] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.059472] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 570.848585] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 570.853359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 570.857525] Lustre: Skipped 1 previous similar message [ 570.867684] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 572.615290] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 575.245530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 581.384036] LustreError: 24804:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 581.388229] LustreError: 24804:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 581.433676] Lustre: server umount lustre-MDT0000 complete [ 583.514031] LustreError: 22462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351840 with bad export cookie 9864446870982018660 [ 583.516381] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.521612] LustreError: 22462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 583.701885] Lustre: server umount lustre-MDT0001 complete [ 589.476092] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:57:25 (1763351845) [ 600.797794] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 606.914179] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 613.961458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 618.597530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 622.924459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 627.754599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 634.516699] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 642.031481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 642.084829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.246107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 642.274668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 642.344470] Lustre: lustre-MDT0000: new disk, initializing [ 642.448943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 645.120304] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 652.219151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 652.260619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 652.327628] Lustre: 28470:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 652.491924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 654.742400] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 660.120667] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 662.448841] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 667.196734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 667.258752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 667.513524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.518342] Lustre: Skipped 4 previous similar messages [ 669.144451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 669.183537] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 669.227722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 669.244600] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 671.209675] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 675.988977] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 681.789171] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 694.754952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 694.758474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.760553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.770171] Lustre: Skipped 3 previous similar messages [ 694.772132] Lustre: Skipped 3 previous similar messages [ 700.208779] Lustre: server umount lustre-OST0000 complete [ 703.973943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.976906] LustreError: Skipped 1 previous similar message [ 708.686290] Lustre: server umount lustre-MDT0000 complete [ 710.113182] LustreError: 29380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.124513] LustreError: 29380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 710.943396] LustreError: 29261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351967 with bad export cookie 9864446870982019059 [ 710.946579] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.948922] LustreError: 29261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 711.125124] LustreError: 31535:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 711.129138] LustreError: 31535:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 711.240323] Lustre: server umount lustre-MDT0001 complete [ 717.315353] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 719.050305] Key type lgssc unregistered [ 719.259680] LNet: 32067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.266756] LNetError: 32067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.278856] LNet: Removed LNI 192.168.201.130@tcp [ 719.858944] Key type .llcrypt unregistered [ 719.861710] Key type ._llcrypt unregistered [ 732.766660] Key type ._llcrypt registered [ 732.768673] Key type .llcrypt registered [ 732.820603] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 733.295691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.321942] alg: No test for adler32 (adler32-zlib) [ 734.228315] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 734.345650] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 735.944446] Key type lgssc registered [ 736.522720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 740.946391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 740.966140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.295036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.772645] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 750.145667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 750.390384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 752.718630] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 756.022444] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 758.157371] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 762.543464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 762.760909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.232171] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 769.182577] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 775.083922] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 782.252665] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:00:38 (1763352038) [ 783.502576] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 784.763673] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 788.620312] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:00:45 (1763352045) [ 794.111865] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:00:50 (1763352050) [ 794.727738] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 800.190743] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 800.196024] Lustre: Skipped 2 previous similar messages [ 801.321637] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:00:57 (1763352057) [ 806.473803] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 806.478193] Lustre: Skipped 4 previous similar messages [ 814.811242] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 814.817448] Lustre: Skipped 5 previous similar messages [ 816.479818] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:01:12 (1763352072) [ 818.208703] Lustre: *** cfs_fail_loc=131b, val=2*** [ 820.235793] Lustre: *** cfs_fail_loc=131b, val=2*** [ 825.001688] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:01:21 (1763352081) [ 832.110272] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 832.113102] Lustre: Skipped 7 previous similar messages [ 833.037172] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:01:29 (1763352089) [ 841.699720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.699816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 841.700818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.706039] Lustre: Skipped 1 previous similar message [ 846.012146] LustreError: 40005:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 846.111436] Lustre: server umount lustre-OST0000 complete [ 851.937076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.947760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.950620] Lustre: Skipped 2 previous similar messages [ 854.424809] LustreError: 40206:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 854.427796] LustreError: 40206:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 854.480386] Lustre: server umount lustre-MDT0000 complete [ 856.616284] LustreError: 33418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352113 with bad export cookie 12082868122852560057 [ 856.618408] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 856.622750] LustreError: 33418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 856.713157] LustreError: 40407:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 856.715093] LustreError: 40407:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 856.821721] Lustre: server umount lustre-MDT0001 complete [ 862.526929] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 864.060317] Key type lgssc unregistered [ 864.239179] LNet: 40938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.243957] LNetError: 40938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.262219] LNet: Removed LNI 192.168.201.130@tcp [ 864.713938] Key type .llcrypt unregistered [ 864.715633] Key type ._llcrypt unregistered [ 875.646829] Key type ._llcrypt registered [ 875.648258] Key type .llcrypt registered [ 875.739862] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 885.452571] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 885.907871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 885.920822] alg: No test for adler32 (adler32-zlib) [ 886.888319] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 887.079794] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 888.703311] Key type lgssc registered [ 889.469754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 895.078272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 899.579941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 903.406573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 907.159467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 911.717820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 911.748394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 911.757699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.926204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 912.966748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 913.036143] Lustre: lustre-MDT0000: new disk, initializing [ 913.097386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.112259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 915.958675] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 922.600907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 922.639000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 922.700829] Lustre: 44045:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 922.740102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 922.743921] Lustre: Skipped 1 previous similar message [ 922.801136] Lustre: lustre-MDT0001: new disk, initializing [ 922.866691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 922.890807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 922.898951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 925.243164] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 929.972586] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 931.694033] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 935.165359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 935.204869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 935.349860] Lustre: lustre-OST0000: new disk, initializing [ 935.353896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 935.363287] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 935.409199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.642879] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 940.552336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 940.560239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 940.584319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 943.843558] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 946.060686] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 946.170570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 948.552901] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 948.667443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 950.752540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 950.756342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.764650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.264231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.273124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.388858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.391775] Lustre: Skipped 3 previous similar messages [ 961.507717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.511593] Lustre: Skipped 1 previous similar message [ 964.063468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 964.091813] LustreError: 46986:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 964.152536] Lustre: server umount lustre-OST0000 complete [ 966.629159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.638041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.644085] Lustre: Skipped 1 previous similar message [ 972.368937] LustreError: 47189:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 972.371333] LustreError: 47189:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 972.433228] Lustre: server umount lustre-MDT0000 complete [ 974.630235] LustreError: 44825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352231 with bad export cookie 407370878493075889 [ 974.634841] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.662279] LustreError: 44825:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 974.774082] LustreError: 47390:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 974.776504] LustreError: 47390:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 974.879675] Lustre: server umount lustre-MDT0001 complete [ 981.248648] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:03:57 (1763352237) [ 985.633202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.956160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.349180] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 991.205699] LustreError: 48110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.217032] LustreError: 48110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 993.733697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 993.945319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.039690] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 999.279864] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1001.688355] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1006.220530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1006.453250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.700466] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1012.944910] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1019.942314] LustreError: 50305:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1020.440374] Lustre: Modifying parameter general.timeout=1 in log params [ 1021.862717] Lustre: Modifying parameter general.timeout=4 in log params [ 1021.866181] Lustre: Skipped 2 previous similar messages [ 1024.066377] Lustre: Modifying parameter general.timeout=9 in log params [ 1024.069183] Lustre: Skipped 4 previous similar messages [ 1028.377859] Lustre: Modifying parameter general.timeout=18 in log params [ 1028.380695] Lustre: Skipped 8 previous similar messages [ 1036.820634] Lustre: Modifying parameter general.timeout=34 in log params [ 1036.823214] Lustre: Skipped 15 previous similar messages [ 1053.105162] Lustre: Modifying parameter general.timeout=69 in log params [ 1053.107501] Lustre: Skipped 34 previous similar messages [ 1089.142902] Lustre: Disabling parameter general.timeout= in log params [ 1089.145483] Lustre: Skipped 60 previous similar messages [ 1090.786763] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:05:47 (1763352347) [ 1092.205157] Lustre: 57217:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1094.186034] Lustre: 57413:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1094.191921] Lustre: 57413:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1097.120663] Lustre: 57727:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1097.126547] Lustre: 57727:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1112.666489] Lustre: 59075:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1112.685145] LustreError: 59075:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1114.970917] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:06:11 (1763352371) [ 1122.273035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1122.276588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.280862] Lustre: Skipped 1 previous similar message [ 1122.283273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.286930] Lustre: Skipped 2 previous similar messages [ 1127.883905] LustreError: 59776:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1127.886093] LustreError: 59776:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1127.935248] Lustre: server umount lustre-MDT0000 complete [ 1129.441329] LustreError: 48105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.455723] LustreError: 48105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1130.084788] LustreError: 48091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352387 with bad export cookie 407370878493076771 [ 1130.087323] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.091364] LustreError: 48091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1130.303624] Lustre: server umount lustre-MDT0001 complete [ 1142.719652] LustreError: 60177:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1142.722712] LustreError: 60177:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1142.814906] Lustre: server umount lustre-OST0000 complete [ 1147.703576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1149.943122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1152.155393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1154.688525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1159.184510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.384862] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1159.404827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1159.410641] Lustre: Skipped 22 previous similar messages [ 1159.574186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.917980] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1167.310267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1167.383133] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1167.405585] Lustre: 61627:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1169.934217] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1173.365399] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1175.345710] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1179.115822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.212492] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1182.421647] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1185.523697] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1191.667778] Lustre: 63863:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1196.516216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.520161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1196.521396] Lustre: Skipped 2 previous similar messages [ 1196.524260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1196.530688] Lustre: Skipped 4 previous similar messages [ 1201.333268] LustreError: 64109:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1201.336300] LustreError: 64109:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1201.395819] Lustre: server umount lustre-OST0000 complete [ 1203.680790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1206.754112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.761955] Lustre: Skipped 1 previous similar message [ 1209.856540] Lustre: server umount lustre-MDT0000 complete [ 1211.873536] LustreError: 61635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.881197] LustreError: 61635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1212.198702] LustreError: 62354:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352469 with bad export cookie 407370878493079333 [ 1212.204701] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1212.385694] Lustre: server umount lustre-MDT0001 complete [ 1218.893787] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1220.529545] Key type lgssc unregistered [ 1220.736244] LNet: 65051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.741545] LNetError: 65051:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.753783] LNet: Removed LNI 192.168.201.130@tcp [ 1221.244623] Key type .llcrypt unregistered [ 1221.246428] Key type ._llcrypt unregistered [ 1234.725889] Key type ._llcrypt registered [ 1234.727348] Key type .llcrypt registered [ 1234.791433] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1235.214523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1235.223242] alg: No test for adler32 (adler32-zlib) [ 1236.165113] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1236.292343] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1237.911548] Key type lgssc registered [ 1238.509482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.841726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.853276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.178854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.602471] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1251.566818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1251.901946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1253.885864] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1256.760366] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1258.716096] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1262.684939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1263.015638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.993090] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1269.255272] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1279.181610] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1286.243248] LustreError: 69129:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1286.312098] Lustre: server umount lustre-MDT0000 complete [ 1286.628699] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.630895] LustreError: 66423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.636318] Lustre: Skipped 2 previous similar messages [ 1286.654647] LustreError: 66423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1288.703426] LustreError: 66404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352545 with bad export cookie 3249126352954291662 [ 1288.710521] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1288.722943] LustreError: 66404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1290.834417] LustreError: 69332:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1290.838658] LustreError: 69332:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1290.966178] Lustre: server umount lustre-MDT0001 complete [ 1303.994095] LustreError: 69534:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1303.996228] LustreError: 69534:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1304.071882] Lustre: server umount lustre-OST0000 complete [ 1308.983711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1311.305561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1313.379851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1315.283648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1318.888322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.056732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1319.082707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1319.244996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.457777] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1326.398755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1326.470841] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1326.489942] Lustre: 70984:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1326.745123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1328.897757] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1331.943910] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1333.817710] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1337.546940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1337.664339] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1337.871639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.646052] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1343.285940] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1344.171249] Lustre: 73223:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1345.111429] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1352.160794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1352.163059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.173721] LustreError: Skipped 1 previous similar message [ 1352.187280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1353.458498] LustreError: 73472:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1353.468198] LustreError: 73472:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1353.550809] Lustre: server umount lustre-OST0000 complete [ 1357.280476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1357.283439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.289192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.292311] Lustre: Skipped 2 previous similar messages [ 1357.296487] Lustre: Skipped 1 previous similar message [ 1362.409149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.413034] Lustre: Skipped 1 previous similar message [ 1367.521551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1367.524729] Lustre: Skipped 1 previous similar message [ 1370.079152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1370.130196] LustreError: 73680:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1370.134566] LustreError: 73680:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1370.184380] Lustre: server umount lustre-MDT0000 complete [ 1372.167146] LustreError: 71714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352629 with bad export cookie 3249126352954292474 [ 1372.170311] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1372.174736] LustreError: 71714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1372.324239] Lustre: server umount lustre-MDT0001 complete [ 1377.412404] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1378.789953] Key type lgssc unregistered [ 1378.952509] LNet: 74413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.956383] LNetError: 74413:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.964742] LNet: Removed LNI 192.168.201.130@tcp [ 1379.390080] Key type .llcrypt unregistered [ 1379.391797] Key type ._llcrypt unregistered [ 1393.193465] Key type ._llcrypt registered [ 1393.197167] Key type .llcrypt registered [ 1393.273906] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1393.833089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.882276] alg: No test for adler32 (adler32-zlib) [ 1394.813928] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1394.958642] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1396.583167] Key type lgssc registered [ 1397.244292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1401.997362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1402.010589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.311417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.519535] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1410.445777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1410.743752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1412.626351] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1415.889717] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1417.905884] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1422.382595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1422.698355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.284706] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1429.687978] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1437.577323] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1438.245692] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1439.557824] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1439.561502] Lustre: Skipped 1 previous similar message [ 1442.102873] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1442.105958] Lustre: Skipped 3 previous similar messages [ 1446.475013] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1446.479416] Lustre: Skipped 6 previous similar messages [ 1454.964721] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1454.967266] Lustre: Skipped 14 previous similar messages [ 1471.415166] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1471.417404] Lustre: Skipped 29 previous similar messages [ 1494.910478] Lustre: 83538:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1496.677148] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:12:33 (1763352753) [ 1497.912372] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1499.143291] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:12:35 (1763352755) [ 1502.689652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1502.692676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.709179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1505.763459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1505.764394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.772839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1507.379349] LustreError: 83834:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1507.447953] Lustre: server umount lustre-OST0000 complete [ 1510.883966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.888409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1510.911791] Lustre: Skipped 1 previous similar message [ 1510.918031] Lustre: Skipped 2 previous similar messages [ 1515.863986] LustreError: 84036:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1515.869688] LustreError: 84036:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1515.935366] Lustre: server umount lustre-MDT0000 complete [ 1516.000310] LustreError: 76504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.007564] LustreError: 76504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1518.108799] LustreError: 78639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352775 with bad export cookie 17555145054440500544 [ 1518.112164] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1518.117143] LustreError: 78639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1518.233351] LustreError: 84237:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1518.236873] LustreError: 84237:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1518.349744] Lustre: server umount lustre-MDT0001 complete [ 1524.640764] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1526.327979] Key type lgssc unregistered [ 1526.488968] LNet: 84768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.493990] LNetError: 84768:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.513497] LNet: Removed LNI 192.168.201.130@tcp [ 1527.015109] Key type .llcrypt unregistered [ 1527.016594] Key type ._llcrypt unregistered [ 1528.983305] Key type ._llcrypt registered [ 1528.985241] Key type .llcrypt registered [ 1529.042665] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1531.198020] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules [ 1531.737252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.809419] alg: No test for adler32 (adler32-zlib) [ 1532.736441] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1532.902716] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1534.543146] Key type lgssc registered [ 1535.248260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1535.958443] LustreError: 85885:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1545.128251] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1549.962585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1556.867580] LustreError: 85885:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1556.889895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.211184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.046742] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1564.348344] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:13:40 (1763352820) [ 1566.741333] LustreError: 86922:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1566.746145] LustreError: 86922:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1566.750790] LustreError: 86922:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1566.831857] LustreError: 87695:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1566.916366] Lustre: server umount lustre-MDT0000 complete [ 1574.091500] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 1575.879805] Key type lgssc unregistered [ 1576.083505] LNet: 88381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.089571] LNetError: 88381:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.105812] LNet: Removed LNI 192.168.201.130@tcp [ 1576.662956] Key type .llcrypt unregistered [ 1576.667780] Key type ._llcrypt unregistered [ 1587.388850] Key type ._llcrypt registered [ 1587.390152] Key type .llcrypt registered [ 1587.458206] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1588.057392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1588.082128] alg: No test for adler32 (adler32-zlib) [ 1589.024344] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1589.165310] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 1590.807236] Key type lgssc registered [ 1591.405838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1595.479715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1595.492642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.796880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.043300] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1602.017832] LustreError: 89560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.025701] LustreError: 89560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1603.106057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1603.419619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1605.354888] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1608.138768] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1609.935882] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1613.495086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1613.851307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.056220] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1620.054980] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1653.754392] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:15:10 (1763352910) [ 1654.972788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1656.264871] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:15:12 (1763352912) [ 1659.872563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1659.875669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.884835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.947956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.949218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.956436] Lustre: Skipped 2 previous similar messages [ 1664.460599] LustreError: 92469:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1664.511131] Lustre: server umount lustre-MDT0000 complete [ 1666.501323] LustreError: 89541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352923 with bad export cookie 17667561188535911086 [ 1666.508362] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1666.608910] LustreError: 92671:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1666.611143] LustreError: 92671:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1666.684260] Lustre: server umount lustre-MDT0001 complete [ 1678.138162] LustreError: 92872:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1678.141732] LustreError: 92872:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1678.283340] Lustre: server umount lustre-OST0000 complete [ 1684.149318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.420433] LustreError: 93501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.559548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.340369] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1689.570854] LustreError: 93501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.799803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1691.174067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1693.020616] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1696.047371] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1697.787475] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1701.090848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1704.827060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1704.873963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1704.962930] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1704.969207] LustreError: 93489:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1704.985862] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1704.990477] LustreError: 95361:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1704.994526] LustreError: 95361:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1704.997647] LustreError: 95361:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1705.039877] Lustre: server umount lustre-OST0000 complete [ 1705.042148] LustreError: 95361:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1709.017772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1709.123396] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1709.127712] LustreError: 93489:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1709.135333] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1709.142081] LustreError: 95823:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 1709.145472] LustreError: 95823:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 1709.148324] LustreError: 95823:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1709.192453] Lustre: server umount lustre-OST0000 complete [ 1709.194295] LustreError: 95823:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1710.744930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1714.138564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1714.242861] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1714.244862] Lustre: Found index 0 for lustre-OST0000, updating log [ 1714.247833] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1714.286809] Lustre: lustre-OST0000: new disk, initializing [ 1714.289481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1714.415267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.337710] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1719.189537] Lustre: Failing over lustre-OST0000 [ 1719.224682] LustreError: 96901:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1719.228476] LustreError: 96901:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1719.281216] Lustre: server umount lustre-OST0000 complete [ 1721.825714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1721.832137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.843728] Lustre: Skipped 1 previous similar message [ 1721.847960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.922822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.924405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.365287] LustreError: 97110:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1727.368191] LustreError: 97110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1727.407838] Lustre: server umount lustre-MDT0000 complete [ 1729.090306] LustreError: 95381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352986 with bad export cookie 17667561188535913830 [ 1729.094900] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.095592] LustreError: 95381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1739.236395] LustreError: 97313:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1739.239432] LustreError: 97313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1739.319920] Lustre: server umount lustre-MDT0001 complete [ 1744.240035] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:16:40 (1763353000) [ 1752.139847] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1757.140502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.339163] LustreError: 98681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1757.388669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.021831] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1762.783944] LustreError: 98681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.969102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1765.047084] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1766.306093] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1769.263138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1769.597355] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1772.018200] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1773.672665] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1773.688791] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1773.688918] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1775.539856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1775.573794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1775.604306] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1775.648535] Lustre: lustre-OST0001: new disk, initializing [ 1775.651105] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1778.296975] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1781.296583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1781.305485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1781.373429] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1787.153244] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1793.302703] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1803.234499] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1803.236456] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.239726] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1803.244856] Lustre: Skipped 2 previous similar messages [ 1803.249440] Lustre: Skipped 2 previous similar messages [ 1806.218361] LustreError: 101920:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1806.221215] LustreError: 101920:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1806.259286] Lustre: server umount lustre-MDT0001 complete [ 1807.647227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1808.353237] LustreError: 99393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1808.359791] LustreError: 99393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1811.139418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1811.201389] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.204314] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1811.226667] Lustre: 98669:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1811.560968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1811.564334] Lustre: Skipped 3 previous similar messages [ 1812.406702] LustreError: 102538:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1812.412548] LustreError: 102538:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1812.417586] LustreError: 98672:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1812.477847] LustreError: 102550:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.130@tcp: cfg command failed: rc = -17 [ 1812.482534] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1812.482534] [ 1813.646457] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1817.064819] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1818.927816] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 23:17:55 (1763353075) [ 1827.266911] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1836.745163] Lustre: 104526:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1848.709057] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1850.433993] Lustre: 105659:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1913.824106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1913.827771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.835143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.838388] Lustre: Skipped 1 previous similar message [ 1917.203990] LustreError: 106115:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1917.206354] LustreError: 106115:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1917.242592] Lustre: server umount lustre-MDT0000 complete [ 1918.813111] LustreError: 101717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353175 with bad export cookie 17667561188535914747 [ 1918.815204] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1918.817446] LustreError: 101717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1918.976430] Lustre: server umount lustre-MDT0001 complete [ 1930.198295] Lustre: server umount lustre-OST0000 complete [ 1942.370124] Lustre: server umount lustre-OST0001 complete [ 1944.755704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1948.717465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1949.113871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1952.670594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1956.667188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1957.241150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1966.971570] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 1971.863906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.872893] Lustre: lustre-MDT0000: reset Object Index mappings [ 1972.093433] LustreError: 109596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.101094] LustreError: 109596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1972.145561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.960705] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1978.267448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1978.275912] Lustre: lustre-MDT0001: reset Object Index mappings [ 1980.478563] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1981.758425] Lustre: 110710:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1984.925894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1988.268776] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1988.456680] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:226 to 0x280000401:257) [ 1992.592214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1994.613745] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1995.463915] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 1997.683492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:354 to 0x280000400:386) [ 1997.687872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2004.868223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2006.600548] Lustre: 112562:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2034.171575] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:21:30 (1763353290) [ 2064.354855] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.354855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.354874] Lustre: Skipped 2 previous similar messages [ 2064.356516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2064.356523] Lustre: Skipped 4 previous similar messages [ 2064.360904] Lustre: Skipped 3 previous similar messages [ 2077.151288] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2077.217170] LustreError: 112915:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2077.219438] LustreError: 112915:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2077.258384] Lustre: server umount lustre-MDT0000 complete [ 2078.963630] LustreError: 109577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353335 with bad export cookie 17667561188535961521 [ 2078.968447] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2078.970574] LustreError: 109577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2079.713035] LustreError: 109591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.719238] LustreError: 109591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2085.349535] Lustre: server umount lustre-MDT0001 complete [ 2093.267726] Lustre: server umount lustre-OST0000 complete [ 2100.987943] Lustre: server umount lustre-OST0001 complete [ 2103.886244] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 2108.304539] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2112.681868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2115.505490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2118.032648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2120.799527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2123.742925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2123.771682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.884666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2123.906905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2123.949862] Lustre: lustre-MDT0000: new disk, initializing [ 2123.997421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.000674] Lustre: Skipped 3 previous similar messages [ 2124.009771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2125.862487] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2129.670846] Lustre: server umount lustre-MDT0000 complete [ 2131.003481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2133.907778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2133.938576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.112737] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2134.121515] Lustre: 116791:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2134.127396] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2136.056568] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2138.582334] Lustre: server umount lustre-MDT0000 complete [ 2142.541268] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:23:19 (1763353399) [ 2143.223389] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2144.039565] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:23:20 (1763353400) [ 2151.207587] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2155.287476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.158710] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2161.119557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2161.152323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2161.197259] Lustre: 118806:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2161.229993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2161.232341] Lustre: Skipped 1 previous similar message [ 2161.275227] Lustre: lustre-MDT0001: new disk, initializing [ 2161.354767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2161.360138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2163.243125] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2166.229158] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2169.294374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2169.341235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2169.466142] Lustre: lustre-OST0000: new disk, initializing [ 2169.469610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2169.586515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2169.590501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2169.671076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2172.275438] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2177.688555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2177.730216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2177.820127] Lustre: lustre-OST0001: new disk, initializing [ 2177.822676] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2179.250876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2179.255178] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2179.310325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2180.647936] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2186.054333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2187.564719] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2369.085686] Lustre: 127586:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2369.088218] Lustre: 127586:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2369.091238] Lustre: 127586:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2369.093996] Lustre: 127586:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2369.096281] Lustre: 127586:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2369.099620] Lustre: 127586:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2503.448301] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:29:19 (1763353759) [ 2517.477396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.480805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2517.483438] Lustre: Skipped 5 previous similar messages [ 2517.487981] Lustre: Skipped 18 previous similar messages [ 2521.609790] LustreError: 127799:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2521.612610] LustreError: 127799:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 2521.690332] Lustre: server umount lustre-MDT0000 complete [ 2522.594642] LustreError: 122007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.603181] LustreError: 122007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2523.340118] LustreError: 118797:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353780 with bad export cookie 17667561188536019544 [ 2523.343586] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.345275] LustreError: 118797:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2529.654778] Lustre: server umount lustre-MDT0001 complete [ 2546.143403] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2546.193301] Lustre: server umount lustre-OST0000 complete [ 2554.055399] Lustre: server umount lustre-OST0001 complete [ 2556.941235] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 2560.930808] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 2565.210824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2567.884406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2570.683127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2573.418219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2576.335675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2576.363426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.464257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2576.480958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2576.522445] Lustre: lustre-MDT0000: new disk, initializing [ 2576.563131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.566081] Lustre: Skipped 5 previous similar messages [ 2576.573219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2578.225355] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2583.579862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2583.608448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2583.653969] Lustre: 130586:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2583.790292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2585.585324] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2589.372832] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2590.923559] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2593.833203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2593.866367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2595.202721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2595.206715] Lustre: Skipped 1 previous similar message [ 2595.209137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2595.257115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2596.353547] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 2599.968892] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2604.218250] Lustre: lustre-MDD0000: changelog on [ 2605.484237] Lustre: lustre-MDD0001: changelog on [ 3400.493970] Lustre: lustre-MDD0001: changelog off [ 3401.339185] Lustre: 131933:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3401.755147] Lustre: lustre-MDD0000: changelog off [ 3404.258991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3404.260426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.267746] Lustre: Skipped 2 previous similar messages [ 3404.271149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3404.273851] Lustre: Skipped 5 previous similar messages [ 3409.714300] LustreError: 140223:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3409.717561] LustreError: 140223:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3409.789690] Lustre: server umount lustre-OST0000 complete [ 3413.472142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3413.475288] LustreError: Skipped 1 previous similar message [ 3413.477828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.480237] Lustre: Skipped 2 previous similar messages [ 3417.770463] Lustre: server umount lustre-MDT0000 complete [ 3419.384560] LustreError: 130577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354676 with bad export cookie 17667561188536820834 [ 3419.389591] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.393381] LustreError: 130577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3419.553479] Lustre: server umount lustre-MDT0001 complete [ 3424.242887] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3425.772702] Key type lgssc unregistered [ 3425.936856] LNet: 141158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3425.940800] LNetError: 141158:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3425.952712] LNet: Removed LNI 192.168.201.130@tcp [ 3426.362143] Key type .llcrypt unregistered [ 3426.363784] Key type ._llcrypt unregistered [ 3434.639242] Key type ._llcrypt registered [ 3434.640623] Key type .llcrypt registered [ 3434.687898] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 3441.597839] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3441.960416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3441.974040] alg: No test for adler32 (adler32-zlib) [ 3442.848197] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3442.947436] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3444.535211] Key type lgssc registered [ 3445.081685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3449.026823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3451.732754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3454.284916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3456.802445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.589670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.615255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3459.622700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.718684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3460.736247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3460.780973] Lustre: lustre-MDT0000: new disk, initializing [ 3460.812144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3460.821705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3462.369631] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3467.254939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.283819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3467.327807] Lustre: 144308:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3467.358910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3467.360888] Lustre: Skipped 1 previous similar message [ 3467.401580] Lustre: lustre-MDT0001: new disk, initializing [ 3467.440758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3467.456326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3467.460665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3469.089516] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3472.824237] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3474.338121] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3477.352843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3477.381121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3477.512186] Lustre: lustre-OST0000: new disk, initializing [ 3477.515856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3477.551698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.105220] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3483.872811] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3486.720524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3486.726376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3486.770115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3491.665678] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_module obdecho/obdecho [ 3492.482312] LustreError: 146930:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3492.486217] LustreError: 146930:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 3493.275903] LustreError: 147026:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 3498.464142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3498.467401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.474409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3502.050173] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.051098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3502.057039] Lustre: Skipped 1 previous similar message [ 3502.061294] Lustre: Skipped 2 previous similar messages [ 3503.303131] LustreError: 147366:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3503.339362] Lustre: server umount lustre-MDT0000 complete [ 3504.880853] LustreError: 145087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354761 with bad export cookie 1100803031300537408 [ 3504.884101] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3504.885037] LustreError: 145087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3504.962110] LustreError: 147569:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3504.965026] LustreError: 147569:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3505.040992] Lustre: server umount lustre-MDT0001 complete [ 3516.992751] LustreError: 147770:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3516.995583] LustreError: 147770:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3517.063501] Lustre: server umount lustre-OST0000 complete [ 3526.055293] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 3530.042391] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3534.385422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3537.092838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3539.533372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3541.945937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.444213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.463031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.554418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3544.570516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3544.609776] Lustre: lustre-MDT0000: new disk, initializing [ 3544.637328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.646145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3546.083631] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3550.626326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.653207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3550.692965] Lustre: 150713:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3550.717970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3550.720159] Lustre: Skipped 1 previous similar message [ 3550.757348] Lustre: lustre-MDT0001: new disk, initializing [ 3550.794793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3550.808130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3550.812307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3552.193080] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3555.501792] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3556.817895] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3559.499294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3559.524327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3559.626167] Lustre: lustre-OST0000: new disk, initializing [ 3559.628689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3559.658772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.510850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3560.514732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3560.555092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3561.822420] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3565.267368] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3566.819131] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3566.910640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3568.445198] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3568.531251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3570.657542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3570.658450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.660272] LustreError: Skipped 1 previous similar message [ 3570.664496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.667334] Lustre: Skipped 1 previous similar message [ 3575.404133] LustreError: 153651:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3575.405996] LustreError: 153651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3575.447604] Lustre: server umount lustre-OST0000 complete [ 3580.896645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.897443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.901727] Lustre: Skipped 2 previous similar messages [ 3586.016103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.018348] Lustre: Skipped 2 previous similar messages [ 3591.135121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3591.173865] LustreError: 153852:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3591.175758] LustreError: 153852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3591.201228] Lustre: server umount lustre-MDT0000 complete [ 3592.464073] LustreError: 150705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354849 with bad export cookie 1100803031300538367 [ 3592.466127] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3592.467242] LustreError: 150705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3592.576495] Lustre: server umount lustre-MDT0001 complete [ 3593.775053] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:47:30 (1763354850) [ 3594.332391] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3594.991625] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:47:31 (1763354851) [ 3597.647176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.835092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.235975] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3602.310265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3603.976974] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3605.991972] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3607.300139] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3609.595866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3611.633349] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3613.427796] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3628.000157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3628.003738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.008865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3628.011273] Lustre: Skipped 2 previous similar messages [ 3632.386315] LustreError: 156992:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3632.389460] LustreError: 156992:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3632.427860] Lustre: server umount lustre-MDT0000 complete [ 3633.632925] LustreError: 154724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.639672] LustreError: 154724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3633.838567] LustreError: 154705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354890 with bad export cookie 1100803031300539235 [ 3633.841780] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3633.843174] LustreError: 154705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3634.186740] LustreError: 156828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3638.751937] LustreError: 156828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3638.752234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.761227] Lustre: Skipped 2 previous similar messages [ 3639.993601] Lustre: server umount lustre-MDT0001 complete [ 3642.520340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3647.824125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3647.932231] LustreError: 158250:0:(llog.c:1616:llog_backup()) MGC192.168.201.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3647.935913] Lustre: 158250:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3658.209401] LustreError: 158258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3658.216284] LustreError: 158258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3663.327877] LustreError: 158258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.567847] LustreError: 158276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.574031] LustreError: 158276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3691.530231] LustreError: 158276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.537382] LustreError: 158276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 3709.407301] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3709.409938] Lustre: 158250:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3723.744067] LustreError: 158258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.751444] LustreError: 158258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 3737.055290] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3737.058648] Lustre: 154715:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3749.863607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3749.866505] Lustre: Skipped 2 previous similar messages [ 3751.512627] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3753.955248] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3754.921628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.087235] LustreError: 154715:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3778.527928] LustreError: 142521:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000014bb6b53 x1849011416964864/t0(0) o250->MGC192.168.201.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3780.239274] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3782.414154] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3783.121646] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3783.653653] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3784.638077] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3788.350496] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3799.009378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3799.009905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.011094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.011100] Lustre: Skipped 7 previous similar messages [ 3799.022356] Lustre: Skipped 1 previous similar message [ 3799.075927] LustreError: 159738:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3799.078644] LustreError: 159738:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3799.119482] Lustre: server umount lustre-MDT0000 complete [ 3800.600165] LustreError: 154703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355057 with bad export cookie 1100803031300545710 [ 3800.604469] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3800.605414] LustreError: 154703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3800.744580] Lustre: server umount lustre-MDT0001 complete [ 3812.513793] Lustre: server umount lustre-OST0000 complete [ 3820.220437] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 3823.624964] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3827.290508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3829.695433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3832.017091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3834.422914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3837.127209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3837.156857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.255669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3837.272809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3837.314113] Lustre: lustre-MDT0000: new disk, initializing [ 3837.349190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.351930] Lustre: Skipped 1 previous similar message [ 3837.360161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3838.890299] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3843.613078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3843.641397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3843.682448] Lustre: 163082:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3843.799526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3845.281153] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3848.825551] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3850.211747] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3852.760495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3852.785046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3855.004841] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3855.613554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3855.660901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3858.235774] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3859.681348] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3859.758558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3861.203140] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3861.293889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3866.082669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3866.082846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.086047] LustreError: Skipped 1 previous similar message [ 3866.090606] Lustre: Skipped 1 previous similar message [ 3866.091434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3866.096287] Lustre: Skipped 3 previous similar messages [ 3876.319102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3876.341100] LustreError: 166018:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3876.343478] LustreError: 166018:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3876.383282] Lustre: server umount lustre-OST0000 complete [ 3879.903936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3892.191292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3892.257703] Lustre: server umount lustre-MDT0000 complete [ 3893.647167] LustreError: 163073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355150 with bad export cookie 1100803031300562531 [ 3893.650308] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3893.651374] LustreError: 163073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3893.775677] Lustre: server umount lustre-MDT0001 complete [ 3896.819648] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:52:33 (1763355153) [ 3899.275245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.427624] LustreError: 167094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.435087] LustreError: 167094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 3900.850077] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3903.967094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3905.566992] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3907.608485] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3908.993662] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3911.540087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3913.821438] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3915.726383] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3929.950930] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:53:06 (1763355186) [ 3935.202208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3935.203212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3935.207268] Lustre: Skipped 3 previous similar messages [ 3935.212053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.214274] Lustre: Skipped 12 previous similar messages [ 3937.037603] Lustre: server umount lustre-OST0000 complete [ 3944.442248] LustreError: 169759:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3944.444104] LustreError: 169759:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3944.467799] Lustre: server umount lustre-MDT0000 complete [ 3945.703416] LustreError: 167073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355202 with bad export cookie 1100803031300563399 [ 3945.707162] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3945.709241] LustreError: 167073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3945.819854] Lustre: server umount lustre-MDT0001 complete [ 3949.391702] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 3950.470593] Key type lgssc unregistered [ 3950.603399] LNet: 170493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3950.606367] LNetError: 170493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3950.613453] LNet: Removed LNI 192.168.201.130@tcp [ 3950.953254] Key type .llcrypt unregistered [ 3950.954227] Key type ._llcrypt unregistered [ 3958.701468] Key type ._llcrypt registered [ 3958.702797] Key type .llcrypt registered [ 3958.737428] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 3959.033885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3959.050088] alg: No test for adler32 (adler32-zlib) [ 3959.906933] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3959.984799] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 3961.567170] Key type lgssc registered [ 3961.974850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3964.623424] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3964.629705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3980.063284] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3985.183140] LustreError: 171704:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.130@tcp: failed processing log, type 4: rc = -110 [ 3990.303367] LustreError: 171704:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3990.307146] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3990.309607] LustreError: MGC192.168.201.130@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3990.314413] LustreError: 171704:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3990.317498] LustreError: 171704:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 3990.320503] LustreError: 171704:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 3990.323234] LustreError: 171704:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3990.345387] Lustre: server umount lustre-OST0000 complete [ 3990.347176] LustreError: 171704:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3993.336560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.498862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.843503] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 3998.805925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3998.950120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4000.309226] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4002.121143] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4003.415144] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4005.970617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4006.065283] LustreError: 173794:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4006.071086] Lustre: 173794:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.130@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4006.126671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.178073] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4010.130646] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4010.880873] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.882074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4010.885489] Lustre: Skipped 1 previous similar message [ 4010.890217] Lustre: Skipped 2 previous similar messages [ 4016.096920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.216332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.218904] Lustre: Skipped 4 previous similar messages [ 4025.311179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4025.345490] LustreError: 174447:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4025.347647] LustreError: 174447:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4025.373902] Lustre: server umount lustre-MDT0000 complete [ 4026.336865] LustreError: 174453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.342775] LustreError: 174453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4026.635217] LustreError: 172187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355283 with bad export cookie 17991251561222134803 [ 4026.637843] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.639872] LustreError: 172187:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4026.699289] LustreError: 174653:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4026.701016] LustreError: 174653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4026.755022] Lustre: server umount lustre-MDT0001 complete [ 4038.320944] LustreError: 174853:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4038.323091] LustreError: 174853:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4038.361839] Lustre: server umount lustre-OST0000 complete [ 4041.408977] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4044.735711] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4048.040986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4050.324287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4052.549146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4054.810864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4057.312306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4057.344025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.443084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4057.455964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4057.495249] Lustre: lustre-MDT0000: new disk, initializing [ 4057.522682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.530205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4058.940850] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4063.272748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4063.290756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4063.326107] Lustre: 177188:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4063.349531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4063.351445] Lustre: Skipped 1 previous similar message [ 4063.387443] Lustre: lustre-MDT0001: new disk, initializing [ 4063.420322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4063.433927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4063.437420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4064.736733] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4067.797194] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4068.924704] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4071.108681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4071.132146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4071.219627] Lustre: lustre-OST0000: new disk, initializing [ 4071.221823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4073.175546] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4073.467548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4073.471175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4073.501296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4076.277337] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4077.620763] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4077.690089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4078.993924] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4079.065979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4081.710074] LustreError: 180126:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4081.712458] LustreError: 180126:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4081.746911] Lustre: server umount lustre-OST0000 complete [ 4083.682189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.687577] Lustre: Skipped 1 previous similar message [ 4083.690214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.691593] Lustre: Skipped 1 previous similar message [ 4088.800559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.802136] Lustre: Skipped 1 previous similar message [ 4092.899218] LustreError: 180326:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4092.901761] LustreError: 180326:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4092.925154] Lustre: server umount lustre-MDT0000 complete [ 4093.920478] LustreError: 177984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.925991] LustreError: 177984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4094.148184] LustreError: 177179:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355351 with bad export cookie 17991251561222135503 [ 4094.151534] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.152513] LustreError: 177179:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4104.723715] Lustre: server umount lustre-MDT0001 complete [ 4107.728663] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:56:04 (1763355364) [ 4113.445614] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4114.464804] Key type lgssc unregistered [ 4114.591396] LNet: 181715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4114.593517] LNetError: 181715:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4114.603313] LNet: Removed LNI 192.168.201.130@tcp [ 4114.882444] Key type .llcrypt unregistered [ 4114.883581] Key type ._llcrypt unregistered [ 4122.135256] Key type ._llcrypt registered [ 4122.136325] Key type .llcrypt registered [ 4122.176543] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4122.596317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.603852] alg: No test for adler32 (adler32-zlib) [ 4123.457306] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4123.542134] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4125.127137] Key type lgssc registered [ 4125.469768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4127.940277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4127.946254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.088222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.233528] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4132.926146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4133.037187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4134.245277] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4135.884467] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4137.059113] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4139.494822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4139.559151] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4139.560689] LustreError: 184525:0:(llog.c:1657:llog_backup()) MGC192.168.201.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4139.563406] Lustre: 184525:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.130@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4139.596500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4141.434718] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4143.176871] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4148.703817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4148.705922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4148.710741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4149.216679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4149.217772] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.223797] Lustre: Skipped 1 previous similar message [ 4150.011575] LustreError: 185181:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4150.035553] Lustre: server umount lustre-MDT0000 complete [ 4151.205149] LustreError: 182871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355408 with bad export cookie 13869807564923925294 [ 4151.208117] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4151.209501] LustreError: 182871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4151.265740] LustreError: 185385:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4151.267345] LustreError: 185385:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4151.311587] Lustre: server umount lustre-MDT0001 complete [ 4162.800960] LustreError: 185585:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4162.803306] LustreError: 185585:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4162.843454] Lustre: server umount lustre-OST0000 complete [ 4165.895478] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4169.171526] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4172.418565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4174.604194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4176.640049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4178.778502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4181.148790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4181.172976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.261161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4181.273700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4181.311659] Lustre: lustre-MDT0000: new disk, initializing [ 4181.335882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.344145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4182.718435] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4187.012868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4187.031486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4187.062886] Lustre: 187918:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4187.079988] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4187.081807] Lustre: Skipped 1 previous similar message [ 4187.083194] LustreError: 188685:0:(llog.c:1657:llog_backup()) MGC192.168.201.130@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4187.087727] LustreError: 188685:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 4187.090736] Lustre: 188685:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.130@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4187.100205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4187.101990] Lustre: Skipped 1 previous similar message [ 4187.133212] Lustre: lustre-MDT0001: new disk, initializing [ 4187.163108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4187.174282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4187.177517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4188.458375] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4191.560874] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4192.733690] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4194.915129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4194.939150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4195.022571] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4195.024150] Lustre: Skipped 1 previous similar message [ 4195.025227] LustreError: 189640:0:(llog.c:1657:llog_backup()) MGC192.168.201.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4195.029192] LustreError: 189640:0:(llog.c:1657:llog_backup()) Skipped 1 previous similar message [ 4195.031549] Lustre: 189640:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.130@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4195.034547] Lustre: 189640:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4195.044738] Lustre: lustre-OST0000: new disk, initializing [ 4195.048418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4196.947460] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4198.390597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4198.394921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4198.426836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4199.903508] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4201.300034] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4201.381479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4202.736290] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4202.817481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4203.488376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4203.489920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4203.490534] LustreError: Skipped 1 previous similar message [ 4203.496303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4203.498760] Lustre: Skipped 3 previous similar messages [ 4208.609951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.612032] Lustre: Skipped 1 previous similar message [ 4209.518081] LustreError: 190854:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4209.520467] LustreError: 190854:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4209.560418] Lustre: server umount lustre-OST0000 complete [ 4212.703779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4212.706067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.710064] Lustre: Skipped 1 previous similar message [ 4212.712054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4216.956897] LustreError: 191055:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4216.958732] LustreError: 191055:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4216.981377] Lustre: server umount lustre-MDT0000 complete [ 4218.208161] LustreError: 188699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355475 with bad export cookie 13869807564923925994 [ 4218.211261] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4218.211862] LustreError: 188699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4218.325178] Lustre: server umount lustre-MDT0001 complete [ 4221.042603] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:57:57 (1763355477) [ 4226.198646] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4229.599663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4229.724807] LustreError: 192622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.731942] LustreError: 192622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4229.738786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.740697] Lustre: Skipped 1 previous similar message [ 4231.012134] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4233.826972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4235.247986] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4236.216872] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4238.339720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4240.217774] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4242.938507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4242.953303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4243.011266] Lustre: lustre-OST0001: new disk, initializing [ 4243.012912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4244.608042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4244.611186] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4244.648983] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4244.944490] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4250.244558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4256.888550] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4262.500836] Lustre: DEBUG MARKER: ADD OST3 [ 4264.234173] loop: module loaded [ 4264.258194] loop0: detected capacity change from 0 to 204800000 [ 4264.387184] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4264.390760] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4264.394753] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4264.448337] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4266.471771] loop0: detected capacity change from 0 to 204800000 [ 4268.342363] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4268.364951] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4268.426080] Lustre: lustre-OST0003: new disk, initializing [ 4268.428135] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4268.466388] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4268.468859] Lustre: Skipped 3 previous similar messages [ 4269.455871] Lustre: DEBUG MARKER: STOP OST3 [ 4270.092931] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4270.146275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4270.150259] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4270.162186] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4270.183446] Lustre: Failing over lustre-OST0003 [ 4270.213118] LustreError: 197007:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 4270.215241] LustreError: 197007:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4270.234348] Lustre: server umount lustre-OST0003 complete [ 4270.687950] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4272.095094] LustreError: 196974:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4273.939508] LustreError: 194266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.673762] Lustre: DEBUG MARKER: START OST3 again [ 4274.954785] LustreError: 194266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.173641] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.176627] Lustre: Skipped 1 previous similar message [ 4275.178278] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4277.033959] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4277.135369] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4278.231196] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4278.583222] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4278.583428] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4279.129082] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4292.658163] Lustre: Failing over lustre-OST0003 [ 4292.681423] LustreError: 198912:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 4292.683146] LustreError: 198912:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4292.702220] Lustre: server umount lustre-OST0003 complete [ 4294.112478] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.112716] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.115456] LustreError: Skipped 1 previous similar message [ 4294.116185] LustreError: 194074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.119486] Lustre: Skipped 2 previous similar messages [ 4294.131370] LustreError: 194074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4302.818975] LustreError: 194072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.824263] LustreError: 194072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4305.888593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4305.891499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4305.893163] Lustre: Skipped 2 previous similar messages [ 4311.530526] Lustre: server umount lustre-MDT0000 complete [ 4312.773486] LustreError: 192603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355569 with bad export cookie 13869807564923926862 [ 4312.778036] LustreError: 192603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4312.778124] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.056673] LustreError: 192646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.057510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.062151] LustreError: 192646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4313.066927] Lustre: Skipped 5 previous similar messages [ 4322.209938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4322.211877] Lustre: Skipped 9 previous similar messages [ 4327.391184] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4327.432126] LustreError: 199384:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 4327.434603] LustreError: 199384:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4327.489915] Lustre: server umount lustre-MDT0001 complete [ 4328.854537] Lustre: server umount lustre-OST0000 complete [ 4330.223371] Lustre: server umount lustre-OST0001 complete [ 4332.512383] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4335.313443] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4339.001425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4341.429686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4343.574728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4345.861964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4348.248808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4348.269062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.352382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4348.368510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4348.408324] Lustre: lustre-MDT0000: new disk, initializing [ 4348.436652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.439716] Lustre: Skipped 1 previous similar message [ 4348.446557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4349.776048] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4354.007615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4354.029622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4354.066416] Lustre: 201971:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4354.182568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4355.559417] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4358.728716] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4359.929896] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4362.291378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4362.311757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4364.412115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4364.456894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4364.541428] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4367.748443] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4369.219229] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4369.295057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4370.680940] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4370.765116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4375.012903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.013724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4375.015022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4375.015026] Lustre: Skipped 2 previous similar messages [ 4375.017173] Lustre: Skipped 1 previous similar message [ 4375.023813] LustreError: Skipped 1 previous similar message [ 4377.561372] Lustre: server umount lustre-OST0000 complete [ 4385.056460] Lustre: server umount lustre-MDT0000 complete [ 4385.248201] LustreError: 202757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.254335] LustreError: 202757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4386.497340] LustreError: 201963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355643 with bad export cookie 13869807564923929116 [ 4386.500880] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4386.503552] LustreError: 201963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4386.623583] Lustre: server umount lustre-MDT0001 complete [ 4387.987258] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:00:44 (1763355644) [ 4393.360251] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4396.580375] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4399.915479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4402.099915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4404.150917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4406.442501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4408.766576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4408.788677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.874632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4408.888234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4408.890405] Lustre: Skipped 3 previous similar messages [ 4408.929142] Lustre: lustre-MDT0000: new disk, initializing [ 4408.931319] Lustre: Skipped 2 previous similar messages [ 4408.962124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4408.964912] Lustre: Skipped 2 previous similar messages [ 4410.363952] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4414.585984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4414.609343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4414.642070] Lustre: 208163:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4414.730900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4414.733890] Lustre: Skipped 3 previous similar messages [ 4414.747778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4416.086913] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4419.298093] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4420.520843] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4422.885354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4422.906959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4424.949030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4425.012308] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4428.199713] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4429.592872] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4429.682755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4431.040591] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4431.129906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4433.838487] LustreError: 211102:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4433.840971] LustreError: 211102:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 4433.878760] Lustre: server umount lustre-OST0000 complete [ 4435.425855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4435.429056] LustreError: Skipped 1 previous similar message [ 4440.544839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.547812] Lustre: Skipped 6 previous similar messages [ 4445.695045] Lustre: server umount lustre-MDT0000 complete [ 4447.050745] LustreError: 208154:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355704 with bad export cookie 13869807564923929998 [ 4447.054478] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4447.056497] LustreError: 208154:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4457.491032] Lustre: server umount lustre-MDT0001 complete [ 4460.444924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.603439] LustreError: 211981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4461.961090] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4464.714581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4466.159548] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4468.023333] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4469.197423] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4471.519040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4473.511379] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4475.281508] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4497.378239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.383968] Lustre: Skipped 5 previous similar messages [ 4503.242383] Lustre: server umount lustre-OST0000 complete [ 4506.079775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4506.081997] LustreError: Skipped 1 previous similar message [ 4510.614599] Lustre: server umount lustre-MDT0000 complete [ 4511.871155] LustreError: 212005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355768 with bad export cookie 13869807564923930880 [ 4511.874770] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.874997] LustreError: 212005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4511.993990] Lustre: server umount lustre-MDT0001 complete [ 4608.784219] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 4609.799904] Key type lgssc unregistered [ 4609.934443] LNet: 215315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4609.938340] LNetError: 215315:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4609.948479] LNet: Removed LNI 192.168.201.130@tcp [ 4610.271147] Key type .llcrypt unregistered [ 4610.272978] Key type ._llcrypt unregistered [ 4618.246668] Key type ._llcrypt registered [ 4618.248087] Key type .llcrypt registered [ 4618.286237] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4624.407905] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4624.750783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4624.773204] alg: No test for adler32 (adler32-zlib) [ 4625.630060] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4625.727310] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 4627.311147] Key type lgssc registered [ 4627.674873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4630.751110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4632.890041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4634.994320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4637.127145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.529117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.548150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4639.553148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.631991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4640.646286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4640.679326] Lustre: lustre-MDT0000: new disk, initializing [ 4640.702935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.708781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4642.078634] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4646.133496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4646.151125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4646.183203] Lustre: 218614:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4646.205201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4646.206750] Lustre: Skipped 1 previous similar message [ 4646.240798] Lustre: lustre-MDT0001: new disk, initializing [ 4646.272812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4646.284754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4646.288566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4647.561171] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4650.615541] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4651.818075] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4654.126781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4654.151022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4654.235931] Lustre: lustre-OST0000: new disk, initializing [ 4654.238324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4654.270678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.182184] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4658.171538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4658.175109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4658.214127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4659.254671] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4660.559265] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4660.637185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4661.835059] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4661.904135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4663.266514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4663.266877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.268450] LustreError: Skipped 1 previous similar message [ 4663.268908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4663.272037] Lustre: Skipped 1 previous similar message [ 4664.558081] LustreError: 221551:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4664.595973] Lustre: server umount lustre-OST0000 complete [ 4666.847686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4666.849686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4666.854963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4666.856243] Lustre: Skipped 1 previous similar message [ 4668.384792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4668.385954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.504316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4673.506592] Lustre: Skipped 2 previous similar messages [ 4678.624497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4678.627265] Lustre: Skipped 1 previous similar message [ 4680.159111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4680.193509] LustreError: 221751:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4680.194960] LustreError: 221751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4680.221714] Lustre: server umount lustre-MDT0000 complete [ 4681.442180] LustreError: 219394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355938 with bad export cookie 10140733999209066441 [ 4681.444824] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4681.446419] LustreError: 219394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4681.503451] LustreError: 221954:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4681.505572] LustreError: 221954:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4681.558777] Lustre: server umount lustre-MDT0001 complete [ 4687.556896] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4691.063207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.224973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.596534] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4695.522694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4695.640509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4696.987652] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4698.141140] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4700.412936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4702.456503] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4705.284334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4705.308725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4705.357902] Lustre: lustre-OST0001: new disk, initializing [ 4705.359440] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4705.387430] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4705.389389] Lustre: Skipped 1 previous similar message [ 4707.228455] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4707.267733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4707.270500] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4707.322322] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4712.590760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4717.188839] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4726.301033] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:06:23 (1763355983) [ 4727.778754] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4727.781543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4727.783621] Lustre: Skipped 1 previous similar message [ 4727.787631] Lustre: Skipped 3 previous similar messages [ 4733.116403] LustreError: 226510:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4733.118269] LustreError: 226510:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4733.145721] Lustre: server umount lustre-MDT0000 complete [ 4734.270118] LustreError: 223847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355991 with bad export cookie 10140733999209067309 [ 4734.273239] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4734.273739] LustreError: 223847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4734.379023] Lustre: server umount lustre-MDT0001 complete [ 4745.201080] LustreError: 226912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4745.203630] LustreError: 226912:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4745.221214] Lustre: server umount lustre-OST0000 complete [ 4756.593900] LustreError: 227114:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4756.595597] LustreError: 227114:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4756.641670] Lustre: server umount lustre-OST0001 complete [ 4758.759099] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 4762.400700] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 4765.859843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4768.159834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4770.352950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4772.668381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4775.109306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4775.134210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.220926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4775.236559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4775.276989] Lustre: lustre-MDT0000: new disk, initializing [ 4775.307501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.315390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4776.704205] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4780.967769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4780.994523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4781.033684] Lustre: 229299:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4781.139118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4782.503690] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4785.723978] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4787.035330] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4789.416767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4789.441232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4789.553777] Lustre: lustre-OST0000: new disk, initializing [ 4789.555897] Lustre: Skipped 1 previous similar message [ 4789.558112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4789.561473] Lustre: Skipped 2 previous similar messages [ 4790.658567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4790.661569] Lustre: Skipped 1 previous similar message [ 4790.663581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4790.710954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4791.655379] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4794.821969] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4796.273418] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4796.351930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4797.808629] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4797.898707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4800.994489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4800.994653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.997600] LustreError: Skipped 1 previous similar message [ 4801.000815] Lustre: Skipped 2 previous similar messages [ 4801.001663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4801.005889] Lustre: Skipped 5 previous similar messages [ 4804.661142] LustreError: 232234:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4804.664733] LustreError: 232234:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4804.709049] Lustre: server umount lustre-OST0000 complete [ 4812.192927] Lustre: server umount lustre-MDT0000 complete [ 4813.523412] LustreError: 229290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356070 with bad export cookie 10140733999209069409 [ 4813.527351] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4813.528214] LustreError: 229290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4813.646660] Lustre: server umount lustre-MDT0001 complete [ 4816.587766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.760332] LustreError: 233112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4816.768302] LustreError: 233112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4816.788492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.791710] Lustre: Skipped 2 previous similar messages [ 4818.237496] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4821.331057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4822.990251] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4824.840927] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4826.113875] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4828.551058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4830.712957] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 4832.665469] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4975.072801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4975.074607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4975.076618] LustreError: Skipped 1 previous similar message [ 4975.079902] Lustre: Skipped 3 previous similar messages [ 4975.080712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4975.084748] Lustre: Skipped 5 previous similar messages [ 4989.407152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4989.427735] LustreError: 235397:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4989.429590] LustreError: 235397:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4989.470598] Lustre: server umount lustre-OST0000 complete [ 4995.554798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4995.554915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.562804] Lustre: Skipped 2 previous similar messages [ 5005.279119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5005.338633] Lustre: server umount lustre-MDT0000 complete [ 5005.792512] LustreError: 233824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.800736] LustreError: 233824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5006.680894] LustreError: 233093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356263 with bad export cookie 10140733999209070291 [ 5006.686301] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5006.686400] LustreError: 233093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5006.819646] Lustre: server umount lustre-MDT0001 complete [ 5096.950914] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5098.054607] Key type lgssc unregistered [ 5098.191340] LNet: 236368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5098.194361] LNetError: 236368:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5098.201269] LNet: Removed LNI 192.168.201.130@tcp [ 5098.528145] Key type .llcrypt unregistered [ 5098.529940] Key type ._llcrypt unregistered [ 5106.581584] Key type ._llcrypt registered [ 5106.583087] Key type .llcrypt registered [ 5106.628478] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5112.546985] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5113.049542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5113.056843] alg: No test for adler32 (adler32-zlib) [ 5113.910771] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5113.985576] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 5115.559143] Key type lgssc registered [ 5115.903591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5118.906322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5121.010840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5123.038300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5125.128437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5130.376859] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5133.906334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5133.924491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5133.929071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.018430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5135.034810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5135.081670] Lustre: lustre-MDT0000: new disk, initializing [ 5135.110237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.118420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5136.465298] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5140.702710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5140.726420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5140.757465] Lustre: 240368:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5140.783362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5140.786922] Lustre: Skipped 1 previous similar message [ 5140.827724] Lustre: lustre-MDT0001: new disk, initializing [ 5140.859925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.871577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5140.875111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5142.212719] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5144.502949] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5146.805255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5146.834797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5146.937624] Lustre: lustre-OST0000: new disk, initializing [ 5146.939558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5146.974301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5148.959060] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5152.759136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5152.762644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5152.801834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5153.244072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5153.273448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5153.333916] Lustre: lustre-OST0001: new disk, initializing [ 5153.335727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5153.369119] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5155.372832] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5160.986242] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5161.988155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5161.991204] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5162.029194] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5165.820589] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5176.799972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5176.803711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5176.808162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5177.313905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5177.315579] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.321137] Lustre: Skipped 2 previous similar messages [ 5180.038642] LustreError: 243894:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5180.072371] Lustre: server umount lustre-MDT0000 complete [ 5181.380430] LustreError: 240360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356438 with bad export cookie 14865975133542077 [ 5181.385420] LustreError: 240360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5181.386046] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.453878] LustreError: 244095:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5181.455937] LustreError: 244095:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5181.513478] Lustre: server umount lustre-MDT0001 complete [ 5192.055067] LustreError: 244296:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5192.059237] LustreError: 244296:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5192.081277] Lustre: server umount lustre-OST0000 complete [ 5203.637384] LustreError: 244497:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5203.639306] LustreError: 244497:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5203.689894] Lustre: server umount lustre-OST0001 complete [ 5208.437661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5211.264126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5211.604875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5214.452400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5217.251856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5217.601752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5223.855138] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5227.225236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.233668] Lustre: lustre-MDT0000: reset Object Index mappings [ 5227.398566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5228.639678] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5231.380200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5231.386257] Lustre: lustre-MDT0001: reset Object Index mappings [ 5232.744877] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5233.668542] Lustre: 248916:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5235.803541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5235.916829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5235.919657] Lustre: Skipped 1 previous similar message [ 5237.820696] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5237.987110] LustreError: 249271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.990972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5237.993543] LustreError: 249271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5240.736152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5241.836928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5242.723130] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5245.823205] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5246.921163] Lustre: 250759:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5257.185695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5257.185976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5257.190117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.193051] Lustre: Skipped 3 previous similar messages [ 5257.194956] Lustre: Skipped 5 previous similar messages [ 5259.328715] LustreError: 250862:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5259.330911] LustreError: 250862:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5259.364161] Lustre: server umount lustre-MDT0000 complete [ 5260.633450] LustreError: 247786:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356517 with bad export cookie 14865975133548853 [ 5260.637303] LustreError: 247786:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5260.637687] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5260.756458] Lustre: server umount lustre-MDT0001 complete [ 5272.501087] LustreError: 251265:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5272.503416] LustreError: 251265:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5272.521955] Lustre: server umount lustre-OST0000 complete [ 5283.155608] Lustre: server umount lustre-OST0001 complete [ 5289.076989] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:15:45 (1763356545) [ 5294.476171] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5297.613526] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5300.736441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5302.818098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5304.923520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5307.155994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5309.483664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5309.507825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.592504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5309.603850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5309.641500] Lustre: lustre-MDT0000: new disk, initializing [ 5309.668624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5309.671152] Lustre: Skipped 1 previous similar message [ 5309.677314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5310.954340] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5315.061303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5315.086833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5315.118294] Lustre: 254979:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5315.226819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5316.524055] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5319.540836] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5320.729868] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5322.951137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5322.975697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5323.068198] Lustre: lustre-OST0000: new disk, initializing [ 5323.069536] Lustre: Skipped 1 previous similar message [ 5323.070909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5323.072565] Lustre: Skipped 2 previous similar messages [ 5324.966595] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5324.993120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5324.997651] Lustre: Skipped 1 previous similar message [ 5325.000106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5325.052847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5327.918991] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5329.154451] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5329.231705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5330.443265] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5330.522250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5335.521422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.522766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5335.525834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5335.526924] LustreError: Skipped 1 previous similar message [ 5335.530104] Lustre: Skipped 2 previous similar messages [ 5337.198116] LustreError: 257920:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5337.200952] LustreError: 257920:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5337.242373] Lustre: server umount lustre-OST0000 complete [ 5340.640757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.641647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5340.645653] Lustre: Skipped 2 previous similar messages [ 5340.650382] Lustre: Skipped 1 previous similar message [ 5344.547382] Lustre: server umount lustre-MDT0000 complete [ 5345.760518] LustreError: 254990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5345.768233] LustreError: 254990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5345.866285] LustreError: 256727:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356602 with bad export cookie 14865975133550939 [ 5345.868813] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5345.870832] LustreError: 256727:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5345.995605] Lustre: server umount lustre-MDT0001 complete [ 5352.210788] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5356.057398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.249899] LustreError: 259495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.272172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.274956] Lustre: Skipped 2 previous similar messages [ 5357.736829] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5360.793832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5362.312375] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5363.327643] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5365.553176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5367.605423] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5370.426788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5370.452728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5370.519236] Lustre: lustre-OST0001: new disk, initializing [ 5370.520879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5371.711187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5371.715757] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5371.751380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5372.513429] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5377.005741] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5378.313446] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5385.296841] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5385.302213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5387.234022] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.235702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5387.238620] Lustre: Skipped 3 previous similar messages [ 5387.242144] Lustre: Skipped 3 previous similar messages [ 5393.344848] LustreError: 263000:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5393.347341] LustreError: 263000:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5393.378543] Lustre: server umount lustre-MDT0000 complete [ 5394.716757] LustreError: 263001:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356651 with bad export cookie 14865975133551828 [ 5394.721929] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5394.722249] LustreError: 263001:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5394.859716] Lustre: server umount lustre-MDT0001 complete [ 5406.723461] Lustre: server umount lustre-OST0000 complete [ 5417.242399] Lustre: server umount lustre-OST0001 complete [ 5422.774669] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5426.516525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5426.672743] LustreError: 264781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5426.696506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5426.699078] Lustre: Skipped 3 previous similar messages [ 5428.101630] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5431.078596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5432.554982] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5433.594800] Lustre: 265889:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5435.893868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5437.027164] LustreError: 266244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5437.032811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5438.042410] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5441.001058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5442.156890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5443.154390] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5446.427256] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5447.784640] Lustre: 267742:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5454.550516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5454.556278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5457.775344] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:18:34 (1763356714) [ 5472.223928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5472.226300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.229913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5472.231751] Lustre: Skipped 4 previous similar messages [ 5475.452377] LustreError: 268447:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5475.454461] LustreError: 268447:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5475.479331] Lustre: server umount lustre-MDT0000 complete [ 5476.645633] LustreError: 264761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356733 with bad export cookie 14865975133558114 [ 5476.648262] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5476.650163] LustreError: 264761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5478.368048] LustreError: 264775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5478.372059] LustreError: 264775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5482.788807] Lustre: server umount lustre-MDT0001 complete [ 5498.335158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5498.377231] Lustre: server umount lustre-OST0000 complete [ 5505.755282] Lustre: server umount lustre-OST0001 complete [ 5507.899714] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5511.399332] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5514.917276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5517.188558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5519.234510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5521.604663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5527.565579] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5531.065551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5531.087441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.169704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5531.183856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5531.219780] Lustre: lustre-MDT0000: new disk, initializing [ 5531.254746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5532.639608] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5536.929193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5536.951371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5536.987054] Lustre: 271929:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5537.092802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5538.493283] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5543.186556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5543.210757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5543.550747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5545.362336] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5549.681780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5549.712626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5551.783247] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5551.807266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5556.390383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5566.963935] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 00:20:23 (1763356823) [ 5567.967824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5567.970176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.974528] Lustre: Skipped 5 previous similar messages [ 5567.976703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5567.979572] Lustre: Skipped 8 previous similar messages [ 5573.791256] Lustre: server umount lustre-MDT0000 complete [ 5575.041092] LustreError: 272709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356832 with bad export cookie 14865975133565989 [ 5575.045610] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5575.046138] LustreError: 272709:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5575.165948] Lustre: server umount lustre-MDT0001 complete [ 5586.882670] Lustre: server umount lustre-OST0000 complete [ 5598.368244] Lustre: server umount lustre-OST0001 complete [ 5604.244013] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5607.445095] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5610.616644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5612.800286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5614.809201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5616.911478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5619.496214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5619.520527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.600318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5619.602866] Lustre: Skipped 2 previous similar messages [ 5619.612849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5619.615639] Lustre: Skipped 4 previous similar messages [ 5619.652334] Lustre: lustre-MDT0000: new disk, initializing [ 5619.654343] Lustre: Skipped 3 previous similar messages [ 5619.677171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.679424] Lustre: Skipped 7 previous similar messages [ 5619.686061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5619.688572] Lustre: Skipped 3 previous similar messages [ 5621.012835] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5625.269094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5625.295208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5625.336530] Lustre: 278981:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5625.445198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5625.447271] Lustre: Skipped 2 previous similar messages [ 5626.748815] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5629.894186] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5631.158194] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5633.567115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5633.590611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5635.054338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5635.691171] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5638.896127] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5639.637771] Lustre: Mounted lustre-client [ 5639.942488] LustreError: 281484:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e0f0a0cb800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 5639.947512] LustreError: 281484:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5639.950165] LustreError: 281484:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 5639.974952] Lustre: Unmounted lustre-client [ 5644.770286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.771530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5644.774070] Lustre: Skipped 4 previous similar messages [ 5644.778720] Lustre: Skipped 5 previous similar messages [ 5645.010571] Lustre: server umount lustre-OST0000 complete [ 5652.380167] Lustre: server umount lustre-MDT0000 complete [ 5653.701429] LustreError: 280726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356910 with bad export cookie 14865975133567515 [ 5653.704470] LustreError: 280726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5653.705068] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5653.827299] Lustre: server umount lustre-MDT0001 complete [ 5657.912779] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5659.017633] Key type lgssc unregistered [ 5691.935326] LNet: 282764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5691.939302] LNetError: 282764:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5768.741320] LNet: Removed LNI 192.168.201.130@tcp [ 5769.055118] Key type .llcrypt unregistered [ 5769.056318] Key type ._llcrypt unregistered [ 5775.146268] Key type ._llcrypt registered [ 5775.147188] Key type .llcrypt registered [ 5775.186432] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 5782.041993] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 5782.371845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5782.422857] alg: No test for adler32 (adler32-zlib) [ 5783.287569] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5783.381992] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 5784.975158] Key type lgssc registered [ 5785.340298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5788.375428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5790.682866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5792.928449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5795.282773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5795.934996] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:24:12 (1763357052) [ 5798.317686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5798.343555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5798.349451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.440600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5799.454215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5799.492953] Lustre: lustre-MDT0000: new disk, initializing [ 5799.519470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.529436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5800.991334] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5805.318645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5805.338816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5805.376484] Lustre: 285936:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5805.404510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5805.406726] Lustre: Skipped 1 previous similar message [ 5805.446620] Lustre: lustre-MDT0001: new disk, initializing [ 5805.483912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5805.500132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5805.503458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5806.855540] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5810.019250] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5811.347819] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5813.728849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5813.752705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5813.854816] Lustre: lustre-OST0000: new disk, initializing [ 5813.857421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5813.889725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.786614] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_default_debug -1 all [ 5818.919925] Lustre: DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5820.408618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5820.412220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5820.440869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5830.627508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5830.628754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.629976] LustreError: Skipped 1 previous similar message [ 5830.635142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5830.636749] Lustre: Skipped 1 previous similar message [ 5834.477106] LustreError: 288381:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5834.524211] Lustre: server umount lustre-OST0000 complete [ 5836.255978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5836.261512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.269436] Lustre: Skipped 1 previous similar message [ 5836.273192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.864421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.864785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.984211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5845.986074] Lustre: Skipped 2 previous similar messages [ 5850.079122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5850.080735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5850.085360] Lustre: Skipped 1 previous similar message [ 5850.119591] LustreError: 288582:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5850.121344] LustreError: 288582:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5850.151047] Lustre: server umount lustre-MDT0000 complete [ 5851.414184] LustreError: 285927:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357108 with bad export cookie 7176355304121206080 [ 5851.417134] LustreError: MGC192.168.201.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5851.417706] LustreError: 285927:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5851.482310] LustreError: 288786:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5851.484566] LustreError: 288786:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5851.535558] Lustre: server umount lustre-MDT0001 complete [ 5855.222184] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5856.295652] Key type lgssc unregistered [ 5856.436417] LNet: 289317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5856.438810] LNetError: 289317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5856.449480] LNet: Removed LNI 192.168.201.130@tcp [ 5856.779442] Key type .llcrypt unregistered [ 5856.780690] Key type ._llcrypt unregistered [ 5891.633334] Key type ._llcrypt registered [ 5891.634335] Key type .llcrypt registered [ 5891.676168] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5891.782243] Key type .llcrypt unregistered [ 5891.784254] Key type ._llcrypt unregistered [ 5919.498270] Key type ._llcrypt registered [ 5919.501253] Key type .llcrypt registered [ 5919.555922] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5919.699279] Key type .llcrypt unregistered [ 5919.700340] Key type ._llcrypt unregistered [ 5930.907936] Key type ._llcrypt registered [ 5930.909036] Key type .llcrypt registered [ 5930.943273] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5931.031332] Key type .llcrypt unregistered [ 5931.032204] Key type ._llcrypt unregistered [ 5942.354681] Key type ._llcrypt registered [ 5942.355699] Key type .llcrypt registered [ 5942.399080] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5942.499405] Key type .llcrypt unregistered [ 5942.500444] Key type ._llcrypt unregistered [ 5970.181118] Key type ._llcrypt registered [ 5970.182132] Key type .llcrypt registered [ 5970.218239] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5970.305170] Key type .llcrypt unregistered [ 5970.306631] Key type ._llcrypt unregistered [ 5981.986379] Key type ._llcrypt registered [ 5981.987290] Key type .llcrypt registered [ 5982.029487] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5982.129199] Key type .llcrypt unregistered [ 5982.130091] Key type ._llcrypt unregistered [ 5993.254645] Key type ._llcrypt registered [ 5993.255592] Key type .llcrypt registered [ 5993.289783] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 5993.384237] Key type .llcrypt unregistered [ 5993.385176] Key type ._llcrypt unregistered [ 6004.148579] Key type ._llcrypt registered [ 6004.149578] Key type .llcrypt registered [ 6004.187914] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6004.281182] Key type .llcrypt unregistered [ 6004.282662] Key type ._llcrypt unregistered [ 6015.084967] Key type ._llcrypt registered [ 6015.085895] Key type .llcrypt registered [ 6015.120308] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6015.215296] Key type .llcrypt unregistered [ 6015.216267] Key type ._llcrypt unregistered [ 6026.121777] Key type ._llcrypt registered [ 6026.123680] Key type .llcrypt registered [ 6026.161551] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6026.259539] Key type .llcrypt unregistered [ 6026.261565] Key type ._llcrypt unregistered [ 6048.802435] Key type ._llcrypt registered [ 6048.803717] Key type .llcrypt registered [ 6048.852791] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6048.989593] Key type .llcrypt unregistered [ 6048.991413] Key type ._llcrypt unregistered [ 6061.005347] Key type ._llcrypt registered [ 6061.006579] Key type .llcrypt registered [ 6061.050472] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6061.157222] Key type .llcrypt unregistered [ 6061.158468] Key type ._llcrypt unregistered [ 6072.433937] Key type ._llcrypt registered [ 6072.434954] Key type .llcrypt registered [ 6072.471652] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6072.557307] Key type .llcrypt unregistered [ 6072.558224] Key type ._llcrypt unregistered [ 6093.196386] Key type ._llcrypt registered [ 6093.197440] Key type .llcrypt registered [ 6093.239596] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing set_hostid [ 6099.881651] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing load_modules_local [ 6100.376910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6100.386731] alg: No test for adler32 (adler32-zlib) [ 6101.251722] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6101.353077] LNet: Added LNI 192.168.201.130@tcp [8/256/0/180] [ 6102.951207] Key type lgssc registered [ 6103.323282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6106.595437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6108.941167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6111.065169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6113.303607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6113.897971] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5858 sec ========== 00:29:30 (1763357370) [ 6114.535064] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:29:31 (1763357371) === [ 6115.701138] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:29:32 (1763357372) === [ 6128.252693] Lustre: DEBUG MARKER: oleg130-server.virtnet: executing unload_modules_local [ 6129.194575] Key type lgssc unregistered [ 6129.309328] LNet: 311008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6129.311334] LNetError: 311008:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6129.322280] LNet: Removed LNI 192.168.201.130@tcp [ 6129.587491] Key type .llcrypt unregistered [ 6129.588497] Key type ._llcrypt unregistered