[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473281935 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002213] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006983] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009000] LSM: Security Framework initializing [ 0.009045] Yama: becoming mindful. [ 0.010029] SELinux: Initializing. [ 0.011057] *** VALIDATE selinux *** [ 0.018768] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023218] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028046] *** VALIDATE tmpfs *** [ 0.029187] *** VALIDATE proc *** [ 0.030126] *** VALIDATE cgroup *** [ 0.030923] *** VALIDATE cgroup2 *** [ 0.031251] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034026] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.038097] debug: unmapping init [mem 0xffffffffabe59000-0xffffffffabe60fff] [ 0.040942] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041577] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042019] ... version: 2 [ 0.042920] ... bit width: 48 [ 0.043013] ... generic registers: 4 [ 0.043876] ... value mask: 0000ffffffffffff [ 0.044013] ... max period: 00007fffffffffff [ 0.045010] ... fixed-purpose events: 3 [ 0.045849] ... event mask: 000000070000000f [ 0.046260] rcu: Hierarchical SRCU implementation. [ 0.048366] smp: Bringing up secondary CPUs ... [ 0.049486] x86: Booting SMP configuration: [ 0.050020] .... node #0, CPUs: #1 #2 #3 [ 0.053649] smp: Brought up 1 node, 4 CPUs [ 0.054916] smpboot: Max logical packages: 1 [ 0.055020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.256953] node 0 deferred pages initialised in 200ms [ 0.260210] devtmpfs: initialized [ 0.261235] x86/mm: Memory block size: 128MB [ 0.263921] gcov: version magic: 0x41383552 [ 0.265340] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.266199] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.267405] pinctrl core: initialized pinctrl subsystem [ 0.268185] [ 0.268542] ************************************************************* [ 0.269014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270015] ** ** [ 0.271013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.272012] ** ** [ 0.273014] ** This means that this kernel is built to expose internal ** [ 0.274014] ** IOMMU data structures, which may compromise security on ** [ 0.275015] ** your system. ** [ 0.276013] ** ** [ 0.277010] ** If you see this message and you are not debugging the ** [ 0.278013] ** kernel, report this immediately to your vendor! ** [ 0.279015] ** ** [ 0.280012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.281013] ************************************************************* [ 0.282619] NET: Registered protocol family 16 [ 0.283438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.284055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.285058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.287117] cpuidle: using governor menu [ 0.288861] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.289618] PCI: Using configuration type 1 for base access [ 0.290170] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.298102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.299045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.301040] cryptd: max_cpu_qlen set to 1000 [ 0.303281] ACPI: Added _OSI(Module Device) [ 0.304015] ACPI: Added _OSI(Processor Device) [ 0.305019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.306016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.310266] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.314288] ACPI: Interpreter enabled [ 0.315073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.316022] ACPI: Using IOAPIC for interrupt routing [ 0.317174] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.318520] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.328062] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.329060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.330029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.331122] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.333582] acpiphp: Slot [2] registered [ 0.334208] acpiphp: Slot [5] registered [ 0.335222] acpiphp: Slot [6] registered [ 0.336174] acpiphp: Slot [7] registered [ 0.337175] acpiphp: Slot [8] registered [ 0.338291] acpiphp: Slot [9] registered [ 0.339186] acpiphp: Slot [10] registered [ 0.340193] acpiphp: Slot [3] registered [ 0.341125] acpiphp: Slot [4] registered [ 0.342153] acpiphp: Slot [11] registered [ 0.343210] acpiphp: Slot [12] registered [ 0.344186] acpiphp: Slot [13] registered [ 0.345149] acpiphp: Slot [14] registered [ 0.346212] acpiphp: Slot [15] registered [ 0.347139] acpiphp: Slot [16] registered [ 0.348142] acpiphp: Slot [17] registered [ 0.349188] acpiphp: Slot [18] registered [ 0.350166] acpiphp: Slot [19] registered [ 0.351150] acpiphp: Slot [20] registered [ 0.352210] acpiphp: Slot [21] registered [ 0.353181] acpiphp: Slot [22] registered [ 0.354157] acpiphp: Slot [23] registered [ 0.355232] acpiphp: Slot [24] registered [ 0.356138] acpiphp: Slot [25] registered [ 0.357165] acpiphp: Slot [26] registered [ 0.358207] acpiphp: Slot [27] registered [ 0.359140] acpiphp: Slot [28] registered [ 0.361145] acpiphp: Slot [29] registered [ 0.363135] acpiphp: Slot [30] registered [ 0.364098] acpiphp: Slot [31] registered [ 0.365164] PCI host bridge to bus 0000:00 [ 0.367050] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.369034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.372034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.375034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.377030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.380042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.382212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.386157] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.389485] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.401023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.405011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.407018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.408025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.411024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.414670] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.417881] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.420052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.425174] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.430024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.444024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.449020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456794] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.463000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.468077] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.499511] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.506021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.514021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.538021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.550506] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.558022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.570025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.596022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.606702] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.613025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.619023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.643027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.659019] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.675021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.689020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.702030] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.709020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.720022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.740024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.764162] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.766341] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.768322] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.770360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.772217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.777167] iommu: Default domain type: Passthrough [ 0.778526] SCSI subsystem initialized [ 0.780184] ACPI: bus type USB registered [ 0.781144] usbcore: registered new interface driver usbfs [ 0.783108] usbcore: registered new interface driver hub [ 0.785085] usbcore: registered new device driver usb [ 0.786209] pps_core: LinuxPPS API ver. 1 registered [ 0.789020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.792063] PTP clock support registered [ 0.794054] EDAC MC: Ver: 3.0.0 [ 0.796084] PCI: Using ACPI for IRQ routing [ 0.797688] NetLabel: Initializing [ 0.799022] NetLabel: domain hash size = 128 [ 0.801019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.804113] NetLabel: unlabeled traffic allowed by default [ 0.806412] vgaarb: loaded [ 0.809376] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.812017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.817186] clocksource: Switched to clocksource kvm-clock [ 0.930415] VFS: Disk quotas dquot_6.6.0 [ 0.932154] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.935090] *** VALIDATE ramfs *** [ 0.936371] *** VALIDATE hugetlbfs *** [ 0.938110] pnp: PnP ACPI init [ 0.940620] pnp: PnP ACPI: found 6 devices [ 0.957802] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.961719] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.964279] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.966781] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.969397] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.972076] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.974875] NET: Registered protocol family 2 [ 0.977781] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.983463] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.987114] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.992486] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.995374] TCP: Hash tables configured (established 65536 bind 65536) [ 0.998153] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.001408] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.004358] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.007234] NET: Registered protocol family 1 [ 1.010118] RPC: Registered named UNIX socket transport module. [ 1.012115] RPC: Registered udp transport module. [ 1.013867] RPC: Registered tcp transport module. [ 1.015552] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.018070] NET: Registered protocol family 44 [ 1.019894] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.022166] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.024347] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.026584] PCI: CLS 0 bytes, default 64 [ 1.028911] Unpacking initramfs... [ 2.467173] debug: unmapping init [mem 0xffff9d573cc54000-0xffff9d573ffbffff] [ 2.473494] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.475418] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.479094] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.958445] Initialise system trusted keyrings [ 2.960169] Key type blacklist registered [ 2.962378] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.971436] zbud: loaded [ 2.974550] *** VALIDATE nfs *** [ 2.975623] *** VALIDATE nfs4 *** [ 2.977116] pstore: using deflate compression [ 2.980488] Platform Keyring initialized [ 3.082857] NET: Registered protocol family 38 [ 3.084402] Key type asymmetric registered [ 3.085723] Asymmetric key parser 'x509' registered [ 3.087123] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.090892] io scheduler mq-deadline registered [ 3.092572] io scheduler kyber registered [ 3.094083] io scheduler bfq registered [ 3.096466] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.099466] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.101914] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.103983] ACPI: Power Button [PWRF] [ 3.109643] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.117210] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.135984] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.145443] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.161187] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.190461] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.223550] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.227839] Non-volatile memory driver v1.3 [ 3.229355] Linux agpgart interface v0.103 [ 3.260710] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.263151] vda: detected capacity change from 0 to 75083776 [ 3.277752] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.280755] vdb: detected capacity change from 0 to 1073741824 [ 3.304365] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.306770] vdc: detected capacity change from 0 to 2621440000 [ 3.321913] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.324357] vdd: detected capacity change from 0 to 2621440000 [ 3.339351] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.342727] vde: detected capacity change from 0 to 4294967296 [ 3.357253] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.360060] vdf: detected capacity change from 0 to 4294967296 [ 3.367479] libphy: Fixed MDIO Bus: probed [ 3.372567] usbcore: registered new interface driver usbserial_generic [ 3.374834] usbserial: USB Serial support registered for generic [ 3.377235] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.381664] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.383703] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.386151] mousedev: PS/2 mouse device common for all mice [ 3.388582] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.390847] rtc_cmos 00:05: RTC can wake from S4 [ 3.394989] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.396285] rtc_cmos 00:05: registered as rtc0 [ 3.400150] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.401187] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.403077] intel_pstate: CPU model not supported [ 3.409535] hid: raw HID events driver (C) Jiri Kosina [ 3.411763] usbcore: registered new interface driver usbhid [ 3.413739] usbhid: USB HID core driver [ 3.415257] drop_monitor: Initializing network drop monitor service [ 3.417378] Initializing XFRM netlink socket [ 3.419278] NET: Registered protocol family 10 [ 3.421847] Segment Routing with IPv6 [ 3.423321] NET: Registered protocol family 17 [ 3.425283] mpls_gso: MPLS GSO support [ 3.430123] RAS: Correctable Errors collector initialized. [ 3.432354] AVX version of gcm_enc/dec engaged. [ 3.434188] AES CTR mode by8 optimization enabled [ 3.508481] sched_clock: Marking stable (3508449446, 0)->(4454910093, -946460647) [ 3.512564] registered taskstats version 1 [ 3.514635] Loading compiled-in X.509 certificates [ 3.516833] zswap: loaded using pool lzo/zbud [ 3.539991] Key type big_key registered [ 3.552745] Key type encrypted registered [ 3.554545] ima: No TPM chip found, activating TPM-bypass! [ 3.556259] ima: Allocated hash algorithm: sha1 [ 3.558068] ima: No architecture policies found [ 3.559365] evm: Initialising EVM extended attributes: [ 3.560764] evm: security.selinux [ 3.561544] evm: security.ima [ 3.562311] evm: security.capability [ 3.563165] evm: HMAC attrs: 0x1 [ 3.565079] rtc_cmos 00:05: setting system clock to 2026-09-05 14:56:15 UTC (1788620175) [ 3.570601] debug: unmapping init [mem 0xfffffffface03000-0xffffffffacffffff] [ 3.572947] debug: unmapping init [mem 0xffffffffabb82000-0xffffffffabe58fff] [ 3.585116] Write protecting the kernel read-only data: 28672k [ 3.588663] debug: unmapping init [mem 0xffffffffaa203000-0xffffffffaa3fffff] [ 3.591435] debug: unmapping init [mem 0xffffffffaab14000-0xffffffffaabfffff] [ 3.625128] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.634133] systemd[1]: Detected virtualization kvm. [ 3.635695] systemd[1]: Detected architecture x86-64. [ 3.637501] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.660406] systemd[1]: No hostname configured. [ 3.662318] systemd[1]: Set hostname to . [ 3.664736] random: systemd: uninitialized urandom read (16 bytes read) [ 3.666956] systemd[1]: Initializing machine ID from random generator. [ 3.713888] random: ln: uninitialized urandom read (6 bytes read) [ 3.799102] random: systemd: uninitialized urandom read (16 bytes read) [ 3.801948] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.808982] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.813752] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.440541] device-mapper: uevent: version 1.0.3 [ 4.442658] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.809661] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.200141] virtio_net virtio0 ens2: renamed from eth0 [ 5.227600] scsi host0: ata_piix [ 5.240293] scsi host1: ata_piix [ 5.242693] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.246043] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.030302] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.067709] random: crng init done [ 10.069121] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.577242] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.766416] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.017455] SELinux: Disabled at runtime. [ 12.084571] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.093292] systemd[1]: Detected virtualization kvm. [ 12.095178] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.666149] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.669761] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.677762] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.682471] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.686484] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.694303] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.698895] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 12.765506] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.095713] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.423152] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.574099] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.649882] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.668085] EDAC sbridge: Ver: 1.1.2 [ 16.296675] Key type dns_resolver registered [ 17.070603] NFS: Registering the id_resolver key type [ 17.073186] Key type id_resolver registered [ 17.075271] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg601-server login: [ 68.899484] hrtimer: interrupt took 3085751 ns [ 85.434649] libcfs: loading out-of-tree module taints kernel. [ 85.527145] Key type ._llcrypt registered [ 85.529262] Key type .llcrypt registered [ 85.627183] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 104.164454] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 105.472831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.484789] alg: No test for adler32 (adler32-zlib) [ 106.916175] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 107.976899] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 109.735156] Key type lgssc registered [ 111.264779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.748850] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.537718] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 177.470168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.517023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.797363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.838575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.909920] Lustre: lustre-MDT0000: new disk, initializing [ 178.974898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.995659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.343304] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.642095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.736382] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.763691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.767841] Lustre: Skipped 1 previous similar message [ 195.858761] Lustre: lustre-MDT0001: new disk, initializing [ 195.932343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.974929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.980962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.283275] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.888414] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.700508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.033504] Lustre: lustre-OST0000: new disk, initializing [ 215.039978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.050565] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.146883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.009143] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.323312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.337256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.502664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.997970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.369444] Lustre: lustre-OST0001: new disk, initializing [ 239.378553] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.384733] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.510154] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.366667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 246.384556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 246.524815] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 247.713624] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.337575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.258801] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.264515] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.947217] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing yml_node [ 285.487317] Lustre: DEBUG MARKER: Client: 2.17.57.85 [ 287.960712] Lustre: DEBUG MARKER: MDS: 2.17.57.85 [ 290.721996] Lustre: DEBUG MARKER: OSS: 2.17.57.85 [ 292.703432] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 11:01:01 EDT 2026 [ 310.148639] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 311.620169] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.190217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 329.197294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.218060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.744637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.745248] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.754847] Lustre: Skipped 2 previous similar messages [ 331.761546] Lustre: Skipped 2 previous similar messages [ 334.039648] Lustre: server umount lustre-MDT0000 complete [ 337.939527] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620509 with bad export cookie 2555326095414735368 [ 337.944475] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.947799] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 341.983874] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.993840] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.004862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.011838] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 342.058250] Lustre: Skipped 2 previous similar messages [ 343.013900] LustreError: 7780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.045098] LustreError: 7780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 344.333572] Lustre: server umount lustre-MDT0001 complete [ 348.066190] Lustre: server umount lustre-OST0000 complete [ 351.737556] Lustre: server umount lustre-OST0001 complete [ 358.002714] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 366.446673] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 401.880902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.224757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.273838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.365613] Lustre: lustre-MDT0000: new disk, initializing [ 402.422555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.444204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.777616] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 416.405861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.513834] Lustre: 15094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 416.584800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 416.589795] Lustre: Skipped 1 previous similar message [ 416.710121] Lustre: lustre-MDT0001: new disk, initializing [ 416.858277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 416.900359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 416.913740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 420.954833] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 430.803739] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 436.321784] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 443.294062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.636973] Lustre: lustre-OST0000: new disk, initializing [ 443.641334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.646695] Lustre: 17064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.716346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.837101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 444.846348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.981249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 449.195468] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 460.773693] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.734904] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.073663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 473.154975] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 473.431296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.742200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.742484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.749428] Lustre: Skipped 1 previous similar message [ 480.758098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.762426] Lustre: Skipped 3 previous similar messages [ 482.244539] Lustre: server umount lustre-OST0000 complete [ 488.418148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 488.424270] LustreError: Skipped 1 previous similar message [ 488.427987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.439691] Lustre: Skipped 1 previous similar message [ 488.445526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.538949] Lustre: server umount lustre-MDT0000 complete [ 496.097357] LustreError: 15107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.947572] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620668 with bad export cookie 2555326095414736901 [ 496.961702] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.507304] Lustre: server umount lustre-MDT0001 complete [ 503.166785] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 11:04:32 (1788620672) [ 520.740125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.233240] LustreError: 19941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.250985] LustreError: 19941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 521.308371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.036074] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 526.308299] LustreError: 19942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.413461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.816198] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 545.714212] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 551.415250] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 553.697084] Lustre: Failing over lustre-MDT0000 [ 554.080469] Lustre: server umount lustre-MDT0000 complete [ 555.495270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 555.503651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.529387] Lustre: Skipped 1 previous similar message [ 555.544771] LustreError: 20669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.578793] LustreError: 20669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 572.897438] Lustre: 3652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788620728/real 1788620728] req@ffff9d57b0df5500 x1875504297836288/t0(0) o400->MGC192.168.206.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788620744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 572.922424] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.962855] LustreError: 20669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.999845] LustreError: 20669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 573.395487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.761844] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 587.770148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 587.811620] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 587.815232] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 597.546644] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 603.105236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 603.113789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.113798] Lustre: Skipped 1 previous similar message [ 603.115151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.146430] Lustre: Skipped 3 previous similar messages [ 608.123987] Lustre: server umount lustre-MDT0000 complete [ 609.255970] LustreError: 19941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.275638] LustreError: 19941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 612.418504] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620784 with bad export cookie 2555326095414738315 [ 612.434484] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.445568] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 612.856816] Lustre: server umount lustre-MDT0001 complete [ 621.780637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.442678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 622.448964] Lustre: Skipped 2 previous similar messages [ 627.292931] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 636.336454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.594386] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 650.491964] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 657.479791] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 660.158876] Lustre: Failing over lustre-MDT0000 [ 660.349169] Lustre: server umount lustre-MDT0000 complete [ 661.224547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 661.241908] Lustre: Skipped 2 previous similar messages [ 676.838136] LustreError: 23278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.859525] LustreError: 23278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 677.856237] Lustre: 3652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788620833/real 1788620833] req@ffff9d57af9f1c00 x1875504297865728/t0(0) o400->MGC192.168.206.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788620849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 677.930568] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.245667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.200419] LustreError: 3648:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d568313aa00 x1875504297868416/t0(0) o250->MGC192.168.206.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 687.686938] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 692.706405] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 692.724073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 692.728268] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 692.731597] Lustre: Skipped 1 previous similar message [ 692.752056] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 704.053734] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 712.162473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.167564] Lustre: Skipped 2 previous similar messages [ 715.256055] Lustre: server umount lustre-MDT0000 complete [ 719.745607] LustreError: 24012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620891 with bad export cookie 2555326095414739253 [ 719.760658] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.763969] LustreError: 24012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 720.280910] Lustre: server umount lustre-MDT0001 complete [ 731.378684] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 11:08:19 (1788620899) [ 748.983661] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 759.791411] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 805.525691] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 816.897499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.079666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 817.151662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 817.278408] Lustre: lustre-MDT0000: new disk, initializing [ 817.438198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 817.449647] Lustre: Skipped 2 previous similar messages [ 817.473886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 822.326358] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 833.736046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.961682] Lustre: 29667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 834.294842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 840.124094] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 851.408408] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 859.004744] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 867.389447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 867.749850] Lustre: 31637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 869.098252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 869.161051] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 869.273448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 869.289866] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 874.663543] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 885.974512] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 893.143966] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 915.429152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 915.440422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.454955] Lustre: Skipped 2 previous similar messages [ 915.459615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 915.470194] Lustre: Skipped 2 previous similar messages [ 917.678622] Lustre: server umount lustre-OST0000 complete [ 928.421986] Lustre: server umount lustre-MDT0000 complete [ 930.787327] LustreError: 29679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.819745] LustreError: 29679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 932.466763] LustreError: 30462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621104 with bad export cookie 2555326095414739659 [ 932.474691] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 932.475718] LustreError: 30462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 933.004144] Lustre: server umount lustre-MDT0001 complete [ 944.870342] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 948.119932] Key type lgssc unregistered [ 948.441801] LNet: 33615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.446888] LNetError: 33615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.459981] LNet: Removed LNI 192.168.206.101@tcp [ 949.581941] Key type .llcrypt unregistered [ 949.587510] Key type ._llcrypt unregistered [ 974.185810] Key type ._llcrypt registered [ 974.190535] Key type .llcrypt registered [ 974.278457] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 975.179169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 975.270438] alg: No test for adler32 (adler32-zlib) [ 976.374649] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 976.723380] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 978.432288] Key type lgssc registered [ 979.492159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 988.287869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 988.312427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.043940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.289685] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 999.395313] LustreError: 35026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.437502] LustreError: 35026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1004.520129] LustreError: 35027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.347955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.065939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1014.089472] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1024.040798] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1031.185617] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1038.714088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.140571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.154547] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1052.163790] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1052.474218] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1059.664263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1076.157623] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 11:14:04 (1788621244) [ 1078.186862] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1079.845870] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1086.755946] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 11:14:15 (1788621255) [ 1096.845328] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 11:14:25 (1788621265) [ 1098.024112] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1107.964637] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1107.971512] Lustre: Skipped 2 previous similar messages [ 1109.945043] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 11:14:38 (1788621278) [ 1113.353856] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1113.358462] Lustre: Skipped 3 previous similar messages [ 1131.469408] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1131.480811] Lustre: Skipped 6 previous similar messages [ 1134.624798] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 11:15:03 (1788621303) [ 1137.409309] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1141.318946] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1150.503167] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:15:19 (1788621319) [ 1152.125164] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1152.138678] Lustre: Skipped 3 previous similar messages [ 1165.391472] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:15:34 (1788621334) [ 1180.644731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1180.657758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.677996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1182.690656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1182.691428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.692063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1184.207633] Lustre: server umount lustre-OST0000 complete [ 1187.809357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.826169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1187.833995] Lustre: Skipped 1 previous similar message [ 1192.416303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1192.426949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.447637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.666390] Lustre: server umount lustre-MDT0000 complete [ 1196.960068] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621368 with bad export cookie 16995089947870523735 [ 1196.970685] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1196.973635] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.381542] Lustre: server umount lustre-MDT0001 complete [ 1205.835624] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 1208.129414] Key type lgssc unregistered [ 1208.385495] LNet: 42901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.390672] LNetError: 42901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.405730] LNet: Removed LNI 192.168.206.101@tcp [ 1209.076680] Key type .llcrypt unregistered [ 1209.079238] Key type ._llcrypt unregistered [ 1223.722205] Key type ._llcrypt registered [ 1223.723907] Key type .llcrypt registered [ 1223.803367] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 1235.873931] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 1236.741436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1236.827736] alg: No test for adler32 (adler32-zlib) [ 1237.847436] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 1238.045942] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 1239.712656] Key type lgssc registered [ 1240.721553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1268.624619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1268.653725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.872626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1269.902298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1269.990488] Lustre: lustre-MDT0000: new disk, initializing [ 1270.058953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.079942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1274.700565] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1285.006234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.156290] Lustre: 46029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1285.193578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1285.196745] Lustre: Skipped 1 previous similar message [ 1285.248353] Lustre: lustre-MDT0001: new disk, initializing [ 1285.329255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1285.377148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1285.400988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1291.123550] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1299.749085] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1305.688559] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1311.434239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.683449] Lustre: lustre-OST0000: new disk, initializing [ 1311.690339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1311.697172] Lustre: 47996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1311.767828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.920898] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1321.572716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1321.588275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1321.791721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1325.336346] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1329.385383] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1329.535300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1333.490870] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1333.742897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1337.313541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1337.317558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.321932] LustreError: Skipped 1 previous similar message [ 1337.322940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1337.331111] Lustre: Skipped 1 previous similar message [ 1341.804435] Lustre: server umount lustre-OST0000 complete [ 1347.040671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1347.049051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.071203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.075290] Lustre: Skipped 1 previous similar message [ 1352.675308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.688024] Lustre: Skipped 2 previous similar messages [ 1357.804542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.811267] Lustre: Skipped 1 previous similar message [ 1359.327864] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1359.543669] Lustre: server umount lustre-MDT0000 complete [ 1362.942831] LustreError: 46022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621534 with bad export cookie 9270391908727919821 [ 1362.950382] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.958126] LustreError: 46022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1363.306349] Lustre: server umount lustre-MDT0001 complete [ 1371.891846] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:19:00 (1788621540) [ 1378.535085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.060019] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1379.080256] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1379.150535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.645815] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1384.418684] LustreError: 50457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.543975] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.517490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.936684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1398.050905] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1405.245218] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1410.498356] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1417.043724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.492986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.465994] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1431.764879] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1436.221723] LustreError: 52983:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1437.095606] Lustre: Modifying parameter general.timeout=1 in log params [ 1438.893653] Lustre: Modifying parameter general.timeout=3 in log params [ 1438.896131] Lustre: Skipped 1 previous similar message [ 1441.707399] Lustre: Modifying parameter general.timeout=6 in log params [ 1441.712308] Lustre: Skipped 2 previous similar messages [ 1446.136672] Lustre: Modifying parameter general.timeout=11 in log params [ 1446.142658] Lustre: Skipped 4 previous similar messages [ 1454.638798] Lustre: Modifying parameter general.timeout=21 in log params [ 1454.644913] Lustre: Skipped 9 previous similar messages [ 1470.890292] Lustre: Modifying parameter general.timeout=41 in log params [ 1470.893687] Lustre: Skipped 19 previous similar messages [ 1503.607384] Lustre: Modifying parameter general.timeout=82 in log params [ 1503.612645] Lustre: Skipped 40 previous similar messages [ 1559.015576] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:22:07 (1788621727) [ 1562.207984] Lustre: 59923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1564.372561] Lustre: 60034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1566.744596] Lustre: 60133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1567.934266] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1567.941402] Lustre: Skipped 52 previous similar messages [ 1571.547392] Lustre: 60330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1571.554359] Lustre: 60330:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1604.262577] Lustre: 61807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1604.283870] Lustre: 61807:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1604.310531] LustreError: 61807:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1608.109359] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:22:57 (1788621777) [ 1623.527807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1623.546501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.559025] Lustre: Skipped 1 previous similar message [ 1623.571428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1625.546832] Lustre: server umount lustre-MDT0000 complete [ 1629.155278] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.194147] LustreError: 50456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1629.490034] LustreError: 53033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621801 with bad export cookie 9270391908727920703 [ 1629.513404] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1630.069102] Lustre: server umount lustre-MDT0001 complete [ 1643.975237] Lustre: server umount lustre-OST0000 complete [ 1670.513567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.854720] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1671.126719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.685454] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1684.899791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.036755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1685.093684] Lustre: 64339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1690.439338] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1699.008044] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1705.216389] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1712.487613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1712.723704] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1713.000316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.007477] Lustre: Skipped 1 previous similar message [ 1719.018505] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1725.910608] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1727.314409] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1727.319970] Lustre: Skipped 18 previous similar messages [ 1727.355126] Lustre: 66909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1735.137621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1735.149392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.171177] Lustre: Skipped 2 previous similar messages [ 1735.181167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1735.189948] Lustre: Skipped 3 previous similar messages [ 1739.918574] Lustre: server umount lustre-OST0000 complete [ 1745.888489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.912354] Lustre: Skipped 1 previous similar message [ 1750.049549] Lustre: server umount lustre-MDT0000 complete [ 1751.013822] LustreError: 64347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1751.030021] LustreError: 64347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1753.395204] LustreError: 66203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621925 with bad export cookie 9270391908727923811 [ 1753.407146] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.408861] LustreError: 66203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1753.734191] Lustre: server umount lustre-MDT0001 complete [ 1765.843896] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 1769.104413] Key type lgssc unregistered [ 1769.333614] LNet: 68118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.337823] LNetError: 68118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.355562] LNet: Removed LNI 192.168.206.101@tcp [ 1770.216934] Key type .llcrypt unregistered [ 1770.219153] Key type ._llcrypt unregistered [ 1794.624530] Key type ._llcrypt registered [ 1794.629288] Key type .llcrypt registered [ 1794.740443] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 1795.821705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.882266] alg: No test for adler32 (adler32-zlib) [ 1796.992197] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 1797.233426] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 1798.871164] Key type lgssc registered [ 1799.872790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1806.455660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1806.471330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.885898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.464870] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1818.086320] LustreError: 69533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.101427] LustreError: 69533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1818.753922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.392959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1823.192802] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1830.815944] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1836.063573] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1843.931475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1844.206646] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1844.622987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.947420] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1858.382511] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1865.330972] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1874.415423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.423856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.428143] Lustre: Skipped 1 previous similar message [ 1874.441049] Lustre: Skipped 2 previous similar messages [ 1879.521734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.534979] Lustre: Skipped 2 previous similar messages [ 1879.987262] Lustre: server umount lustre-MDT0000 complete [ 1884.468419] LustreError: 69514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622056 with bad export cookie 18183278761608055153 [ 1884.471930] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1884.484628] LustreError: 69514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1884.640730] LustreError: 69529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1884.641048] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.654745] Lustre: Skipped 1 previous similar message [ 1884.714456] Lustre: server umount lustre-MDT0001 complete [ 1890.961624] Lustre: server umount lustre-OST0000 complete [ 1915.555231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.799159] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1915.825243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1916.046451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.950376] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1927.065188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.128306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1927.151392] Lustre: 74411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1927.454221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.454701] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1938.713125] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1943.747923] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1950.899243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1951.160142] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1951.522180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.663504] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 1964.693222] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1966.324963] Lustre: 76985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1968.084298] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1973.729927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.730437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.745327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.336936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1978.350812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.369448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.843256] Lustre: server umount lustre-OST0000 complete [ 1983.970974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.978289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1989.090405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1989.098102] Lustre: Skipped 4 previous similar messages [ 1998.305019] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1998.531616] Lustre: server umount lustre-MDT0000 complete [ 2001.902628] LustreError: 74404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622173 with bad export cookie 18183278761608055993 [ 2001.909067] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.917567] LustreError: 74404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2002.215898] Lustre: server umount lustre-MDT0001 complete [ 2012.358881] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 2014.838204] Key type lgssc unregistered [ 2015.175987] LNet: 78200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.182995] LNetError: 78200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.208760] LNet: Removed LNI 192.168.206.101@tcp [ 2015.895675] Key type .llcrypt unregistered [ 2015.897805] Key type ._llcrypt unregistered [ 2039.686589] Key type ._llcrypt registered [ 2039.689697] Key type .llcrypt registered [ 2039.815586] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 2041.308686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2041.335024] alg: No test for adler32 (adler32-zlib) [ 2042.397172] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 2042.675438] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 2044.367351] Key type lgssc registered [ 2045.472964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2052.518837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2052.546945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.030096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.264723] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2063.331408] LustreError: 79611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.349049] LustreError: 79611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2066.171233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.754932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.952243] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2077.843339] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.424083] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2089.641776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2090.177553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.732480] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2103.876901] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2110.027894] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2110.910751] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2112.483911] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2112.489091] Lustre: Skipped 1 previous similar message [ 2115.217905] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2115.222937] Lustre: Skipped 2 previous similar messages [ 2120.136838] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2120.140603] Lustre: Skipped 4 previous similar messages [ 2128.346780] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2128.353906] Lustre: Skipped 8 previous similar messages [ 2144.353182] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2144.359560] Lustre: Skipped 17 previous similar messages [ 2176.537338] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2176.544092] Lustre: Skipped 37 previous similar messages [ 2202.173248] Lustre: 87710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2204.701258] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:32:53 (1788622373) [ 2206.278914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2208.116694] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:32:57 (1788622377) [ 2214.370491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2214.383805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.394604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.416800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2216.432567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.433225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.463157] Lustre: Skipped 1 previous similar message [ 2217.911109] Lustre: server umount lustre-OST0000 complete [ 2225.122542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2225.131598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.145911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.992637] Lustre: server umount lustre-MDT0000 complete [ 2231.558563] LustreError: 83287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622403 with bad export cookie 5235268091853325400 [ 2231.563155] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2231.569269] LustreError: 83287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2231.894480] Lustre: server umount lustre-MDT0001 complete [ 2242.118787] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 2244.862179] Key type lgssc unregistered [ 2245.145836] LNet: 88957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.159651] LNetError: 88957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2245.212242] LNet: Removed LNI 192.168.206.101@tcp [ 2246.086810] Key type .llcrypt unregistered [ 2246.091565] Key type ._llcrypt unregistered [ 2250.477820] Key type ._llcrypt registered [ 2250.481095] Key type .llcrypt registered [ 2250.627459] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2255.467616] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules [ 2256.714538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2256.833417] alg: No test for adler32 (adler32-zlib) [ 2257.843475] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 2258.078479] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 2259.767557] Key type lgssc registered [ 2261.089354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2262.323269] LustreError: 90164:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2277.725550] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 2283.592670] LustreError: 90164:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2286.970202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2286.987281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.448064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.928992] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2298.854040] LustreError: 91181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.880103] LustreError: 91181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2301.390685] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:34:30 (1788622470) [ 2303.973627] LustreError: 91182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.395349] LustreError: 91213:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2306.417549] LustreError: 91213:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2306.423802] LustreError: 91213:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2306.716720] Lustre: server umount lustre-MDT0000 complete [ 2320.643650] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 2323.018598] Key type lgssc unregistered [ 2323.307959] LNet: 92705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.314345] LNetError: 92705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.329831] LNet: Removed LNI 192.168.206.101@tcp [ 2324.180296] Key type .llcrypt unregistered [ 2324.183281] Key type ._llcrypt unregistered [ 2341.317814] Key type ._llcrypt registered [ 2341.320365] Key type .llcrypt registered [ 2341.442115] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 2342.465084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.485078] alg: No test for adler32 (adler32-zlib) [ 2343.552229] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 2343.744507] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 2345.383170] Key type lgssc registered [ 2346.477940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2353.662923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2353.690993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.218369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.843190] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2365.422989] LustreError: 93921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.459820] LustreError: 93921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2368.711246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.423955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2373.753991] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2382.010937] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2387.647828] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2394.153686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2394.847578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.558718] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2407.195391] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2455.888398] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:37:04 (1788622624) [ 2457.346694] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2459.013485] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:37:08 (1788622628) [ 2464.745171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.756603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.758062] Lustre: Skipped 1 previous similar message [ 2464.767082] Lustre: Skipped 2 previous similar messages [ 2468.890529] Lustre: server umount lustre-MDT0000 complete [ 2469.879156] LustreError: 93917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.905559] LustreError: 93917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2472.497098] LustreError: 94660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622644 with bad export cookie 9433266777422038232 [ 2472.498170] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.504374] LustreError: 94660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2472.833288] Lustre: server umount lustre-MDT0001 complete [ 2486.919155] Lustre: server umount lustre-OST0000 complete [ 2498.049095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.530827] LustreError: 98191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.746806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.694616] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2504.163162] LustreError: 98192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.261058] LustreError: 98191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.347402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.102228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2518.228794] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2527.151356] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2534.360931] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2549.000227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2549.268403] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2549.275696] LustreError: 98181:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2549.311310] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2549.319885] LustreError: 100275:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2549.326327] LustreError: 100275:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2549.333020] LustreError: 100275:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2549.431741] Lustre: server umount lustre-OST0000 complete [ 2549.445357] LustreError: 100275:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2557.326633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.479502] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2557.484828] LustreError: 98178:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2557.491330] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2557.501825] LustreError: 100737:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2557.506733] LustreError: 100737:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2557.510627] LustreError: 100737:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2557.566069] Lustre: server umount lustre-OST0000 complete [ 2557.568267] LustreError: 100737:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2565.805937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2566.050729] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2566.059602] Lustre: Found index 0 for lustre-OST0000, updating log [ 2566.065657] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2566.143791] Lustre: lustre-OST0000: new disk, initializing [ 2566.149264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2566.158910] Lustre: 101272:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2566.560488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.744775] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2576.201089] Lustre: Failing over lustre-OST0000 [ 2576.352920] Lustre: server umount lustre-OST0000 complete [ 2582.821199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2582.833764] Lustre: Skipped 1 previous similar message [ 2582.840585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2582.845130] Lustre: Skipped 1 previous similar message [ 2588.128830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.134241] Lustre: Skipped 1 previous similar message [ 2593.247954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.261088] Lustre: Skipped 1 previous similar message [ 2594.271162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2594.480207] Lustre: server umount lustre-MDT0000 complete [ 2597.814567] LustreError: 98171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622769 with bad export cookie 9433266777422040976 [ 2597.825914] LustreError: 98171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2597.826700] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.354493] Lustre: server umount lustre-MDT0001 complete [ 2605.739750] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:39:34 (1788622774) [ 2620.880455] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 2631.301671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.711884] LustreError: 103618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.799623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.277918] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2644.504826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.796898] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2652.577857] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2658.962807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.671839] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2665.443770] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2669.941068] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2669.949952] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2669.961462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2673.523495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.598982] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2673.688890] Lustre: lustre-OST0001: new disk, initializing [ 2673.694194] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2673.699753] Lustre: 105940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2673.906150] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2673.927100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2673.942544] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2673.946880] Lustre: Skipped 2 previous similar messages [ 2677.044043] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 2677.058521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2677.087707] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2677.087943] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2677.113240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2679.378519] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2679.924526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2679.941312] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2680.036213] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2686.428255] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2692.318682] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2696.675564] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2696.684525] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.700220] Lustre: Skipped 1 previous similar message [ 2696.713544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2700.774178] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.798835] Lustre: Skipped 1 previous similar message [ 2702.570562] Lustre: server umount lustre-MDT0001 complete [ 2705.894626] LustreError: 104352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.913940] LustreError: 104352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2710.634967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.722398] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2710.726319] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2710.790140] Lustre: 103607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2712.519378] LustreError: 107508:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2712.533827] LustreError: 107508:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2712.553320] LustreError: 103610:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2712.691928] LustreError: 107522:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.101@tcp: cfg command failed: rc = -17 [ 2712.699574] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2712.699574] [ 2715.620508] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2716.665946] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2725.843525] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:41:34 (1788622894) [ 2744.432764] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 2761.054576] Lustre: 109552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2784.183224] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2787.755257] Lustre: 110687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2864.609749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2864.617126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.629707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.632659] Lustre: Skipped 5 previous similar messages [ 2870.337590] Lustre: server umount lustre-MDT0000 complete [ 2873.765964] LustreError: 103599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623045 with bad export cookie 9433266777422041907 [ 2873.774266] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.777142] LustreError: 103599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2874.063750] Lustre: server umount lustre-MDT0001 complete [ 2887.427284] Lustre: server umount lustre-OST0000 complete [ 2901.630285] Lustre: server umount lustre-OST0001 complete [ 2907.091123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2916.940224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2925.045465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2934.603270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2955.666365] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 2967.176291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.201595] Lustre: lustre-MDT0000: reset Object Index mappings [ 2967.735236] LustreError: 114643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.764676] LustreError: 114643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2967.894749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.908913] Lustre: Skipped 1 previous similar message [ 2974.151311] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2982.727824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.748455] Lustre: lustre-MDT0001: reset Object Index mappings [ 2987.475084] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 2990.348780] Lustre: 115791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2996.142102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3002.090861] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 3009.011929] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3009.631104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3012.298391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3012.305120] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3012.324142] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3016.286110] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 3023.850521] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3027.416053] Lustre: 117666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3072.380318] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:47:21 (1788623241) [ 3099.624717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.634594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.637896] Lustre: Skipped 5 previous similar messages [ 3099.650884] Lustre: Skipped 11 previous similar messages [ 3105.076484] Lustre: server umount lustre-MDT0000 complete [ 3108.234075] LustreError: 114624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623280 with bad export cookie 9433266777422088982 [ 3108.241528] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.243182] LustreError: 114624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3109.867353] LustreError: 117923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.894542] LustreError: 117923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3114.872810] Lustre: server umount lustre-MDT0001 complete [ 3125.638302] Lustre: server umount lustre-OST0000 complete [ 3144.159499] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3144.423206] Lustre: server umount lustre-OST0001 complete [ 3150.575710] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 3160.052875] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 3191.113346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.288025] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3191.307952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3191.366679] Lustre: lustre-MDT0000: new disk, initializing [ 3191.417559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.421406] Lustre: Skipped 3 previous similar messages [ 3191.437543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3195.232863] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 3201.023270] Lustre: server umount lustre-MDT0000 complete [ 3209.405246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.770357] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3209.782588] Lustre: 121939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3209.793192] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3214.593178] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 3220.314631] Lustre: server umount lustre-MDT0000 complete [ 3227.872706] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:49:57 (1788623397) [ 3229.210373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3230.600884] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:50:00 (1788623400) [ 3246.326238] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 3254.085197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.473828] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 3266.639616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.757395] Lustre: 123986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3266.865420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3266.873038] Lustre: Skipped 1 previous similar message [ 3267.003362] Lustre: lustre-MDT0001: new disk, initializing [ 3267.153990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3267.179837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3271.819954] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 3276.054954] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3281.541643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3281.816451] Lustre: lustre-OST0000: new disk, initializing [ 3281.822866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3281.828558] Lustre: 125566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3283.588980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3283.599574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3283.732601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3287.032559] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 3296.117198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3296.276291] Lustre: lustre-OST0001: new disk, initializing [ 3296.283565] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3296.292624] Lustre: 126435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3297.689376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3297.694306] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3297.779646] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3301.301640] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 3309.253597] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3312.198807] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3665.757605] Lustre: 132845:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3665.766566] Lustre: 132845:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3665.773261] Lustre: 132845:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3665.779217] Lustre: 132845:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3665.784391] Lustre: 132845:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3665.789689] Lustre: 132845:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3912.052160] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 12:01:20 (1788624080) [ 3927.521533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3927.526743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3927.534681] Lustre: Skipped 3 previous similar messages [ 3927.537825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3927.546527] Lustre: Skipped 8 previous similar messages [ 3930.602200] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.626883] Lustre: Skipped 1 previous similar message [ 3932.794356] Lustre: server umount lustre-MDT0000 complete [ 3935.719904] LustreError: 127240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3935.745806] LustreError: 127240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3936.644601] LustreError: 132870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624108 with bad export cookie 9433266777422155041 [ 3936.651418] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3936.663439] LustreError: 132870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3940.835319] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.853976] Lustre: Skipped 1 previous similar message [ 3940.862996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3940.870777] Lustre: Skipped 5 previous similar messages [ 3943.203246] Lustre: server umount lustre-MDT0001 complete [ 3961.311192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3961.551854] Lustre: server umount lustre-OST0000 complete [ 3979.231181] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3979.426630] Lustre: server umount lustre-OST0001 complete [ 3985.784335] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 3994.093747] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 4026.835208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4027.092480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4027.118151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4027.196977] Lustre: lustre-MDT0000: new disk, initializing [ 4027.272400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4027.274536] Lustre: Skipped 5 previous similar messages [ 4027.291095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4031.347993] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 4040.288649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.420650] Lustre: 135903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4040.476775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4040.484342] Lustre: Skipped 1 previous similar message [ 4040.623851] Lustre: lustre-MDT0001: new disk, initializing [ 4040.789123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4040.801452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4044.737813] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 4052.419386] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4057.172237] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4063.034374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4063.306771] Lustre: lustre-OST0000: new disk, initializing [ 4063.310393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4063.316585] Lustre: 137868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4065.137099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4065.156549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4065.320749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4069.693308] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 4078.913300] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4087.456455] Lustre: lustre-MDD0000: changelog on [ 4090.322076] Lustre: lustre-MDD0001: changelog on [ 5697.489833] Lustre: lustre-MDD0001: changelog off [ 5699.305864] Lustre: 139321:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5700.146237] Lustre: lustre-MDD0000: changelog off [ 5708.774709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.787192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.795478] LustreError: Skipped 1 previous similar message [ 5708.797939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.797952] Lustre: Skipped 2 previous similar messages [ 5708.844461] Lustre: Skipped 2 previous similar messages [ 5712.664813] Lustre: server umount lustre-OST0000 complete [ 5719.009872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.029666] Lustre: Skipped 1 previous similar message [ 5719.032362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5719.042504] Lustre: Skipped 1 previous similar message [ 5722.916782] Lustre: server umount lustre-MDT0000 complete [ 5724.130636] LustreError: 137064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.166164] LustreError: 137064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5727.478409] LustreError: 135896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625899 with bad export cookie 9433266777422974342 [ 5727.482302] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5727.497822] LustreError: 135896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5728.188580] Lustre: server umount lustre-MDT0001 complete [ 5740.604910] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 5743.360713] Key type lgssc unregistered [ 5743.730907] LNet: 146950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5743.741342] LNetError: 146950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5743.760434] LNet: Removed LNI 192.168.206.101@tcp [ 5744.569352] Key type .llcrypt unregistered [ 5744.580074] Key type ._llcrypt unregistered [ 5762.390769] Key type ._llcrypt registered [ 5762.397156] Key type .llcrypt registered [ 5762.507250] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 5777.569927] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 5778.462521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5778.507588] alg: No test for adler32 (adler32-zlib) [ 5779.595153] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 5779.904961] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 5781.610133] Key type lgssc registered [ 5782.791885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5817.444574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5817.469719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.692593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5818.712444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5818.771541] Lustre: lustre-MDT0000: new disk, initializing [ 5818.881603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5818.921774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5823.427941] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 5834.689668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5834.839791] Lustre: 150146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5834.898292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5834.902791] Lustre: Skipped 1 previous similar message [ 5835.015463] Lustre: lustre-MDT0001: new disk, initializing [ 5835.189781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5835.263578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5835.298263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5839.436298] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 5848.358441] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5853.299236] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5859.714911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5860.011877] Lustre: lustre-OST0000: new disk, initializing [ 5860.018191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5860.027185] Lustre: 152118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5860.116758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5865.895734] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 5869.656361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5869.673661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5869.826571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5874.136026] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5879.944560] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_module obdecho/obdecho [ 5881.392260] LustreError: 153122:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5881.398534] LustreError: 153122:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5882.830564] LustreError: 153218:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5895.649566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.653640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5895.671741] Lustre: Skipped 2 previous similar messages [ 5895.683926] Lustre: Skipped 2 previous similar messages [ 5898.525707] Lustre: server umount lustre-MDT0000 complete [ 5902.138923] LustreError: 150141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626074 with bad export cookie 9318701599451926548 [ 5902.146871] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.152108] LustreError: 150141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5902.517152] Lustre: server umount lustre-MDT0001 complete [ 5916.605856] Lustre: server umount lustre-OST0000 complete [ 5935.789332] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 5946.170984] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 5979.635615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.797523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5979.835759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5979.922089] Lustre: lustre-MDT0000: new disk, initializing [ 5980.014820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5980.032174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5983.405364] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 5991.718793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.841333] Lustre: 156939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5991.900410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5991.906269] Lustre: Skipped 1 previous similar message [ 5991.983681] Lustre: lustre-MDT0001: new disk, initializing [ 5992.070871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5992.096617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5992.107652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5996.202959] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6004.408813] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6009.525452] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6015.928825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6016.208221] Lustre: lustre-OST0000: new disk, initializing [ 6016.214341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6016.223108] Lustre: 158904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6016.309613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.875398] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6023.256180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6023.262429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6023.371656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6030.221445] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6034.318570] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6034.499918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6038.834540] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6039.029960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6044.130722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.134582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.135390] LustreError: Skipped 1 previous similar message [ 6044.138189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6044.154233] Lustre: Skipped 1 previous similar message [ 6049.251790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6049.259336] Lustre: Skipped 2 previous similar messages [ 6054.373713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6054.386538] Lustre: Skipped 1 previous similar message [ 6055.395347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6055.583524] Lustre: server umount lustre-OST0000 complete [ 6058.980604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6058.986455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.995585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6069.731857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6069.737207] Lustre: Skipped 5 previous similar messages [ 6073.311124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6073.505840] Lustre: server umount lustre-MDT0000 complete [ 6074.849143] LustreError: 156950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.869944] LustreError: 156950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6076.554850] LustreError: 158903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626248 with bad export cookie 9318701599451927507 [ 6076.556258] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6076.563994] LustreError: 158903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6076.856760] Lustre: server umount lustre-MDT0001 complete [ 6080.082830] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:37:29 (1788626249) [ 6081.405838] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6082.966473] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:37:32 (1788626252) [ 6088.898920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.232307] LustreError: 161318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.282885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6092.832938] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6094.306448] LustreError: 161319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.431297] LustreError: 161318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.581868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.607133] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6111.135657] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6116.256576] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6123.171206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.483224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.490633] Lustre: Skipped 1 previous similar message [ 6129.311984] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6136.359093] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6142.160825] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.1@tcp (stopping) [ 6142.499858] Lustre: server umount lustre-MDT0000 complete [ 6144.995577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6144.999487] LustreError: 161318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.004879] Lustre: Skipped 3 previous similar messages [ 6145.021290] LustreError: 161318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6146.440227] LustreError: 161300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626318 with bad export cookie 9318701599451928389 [ 6146.451185] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.454833] LustreError: 161300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6150.114967] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.810371] Lustre: server umount lustre-MDT0001 complete [ 6158.990484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6171.503906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.761850] LustreError: 165158:0:(llog.c:1655:llog_backup()) MGC192.168.206.101@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6171.771468] Lustre: 165158:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.101@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6176.739508] LustreError: 165163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.758918] LustreError: 165163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6193.317090] LustreError: 165181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6193.342812] LustreError: 165181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6226.912857] LustreError: 165165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6226.927843] LustreError: 165165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 6232.547799] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6232.567789] Lustre: 165158:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.101@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6273.526544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6277.125860] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6278.632900] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6284.731489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.951300] LustreError: 163149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6291.954157] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8152adbb6b32707b [ 6291.966633] LustreError: 163149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6291.994260] Lustre: MGC192.168.206.101@tcp: Connection restored to 0@lo (at 0@lo) [ 6296.192617] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6297.585988] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6303.169860] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6304.570055] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6309.065861] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6310.232742] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6328.290829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6328.291929] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.300350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6328.309123] Lustre: Skipped 2 previous similar messages [ 6328.324610] Lustre: Skipped 6 previous similar messages [ 6332.764706] Lustre: server umount lustre-MDT0000 complete [ 6335.500435] LustreError: 161298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626507 with bad export cookie 9318701599451934843 [ 6335.503098] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6335.512044] LustreError: 161298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6335.816028] Lustre: server umount lustre-MDT0001 complete [ 6349.423706] Lustre: server umount lustre-OST0000 complete [ 6370.799456] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 6380.764655] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 6411.464982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.679775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6411.700817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6411.761608] Lustre: lustre-MDT0000: new disk, initializing [ 6411.889532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.894509] Lustre: Skipped 1 previous similar message [ 6411.903974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6416.087449] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6425.367950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.466234] Lustre: 170265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6425.731498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6429.213625] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6437.715312] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6442.061457] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6447.199297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6447.423361] Lustre: lustre-OST0000: new disk, initializing [ 6447.425807] Lustre: Skipped 1 previous similar message [ 6447.428633] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6447.435144] Lustre: Skipped 2 previous similar messages [ 6447.440418] Lustre: 172232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6452.994717] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6454.835115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6454.851178] Lustre: Skipped 1 previous similar message [ 6454.856626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6454.942218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6461.404721] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6465.620523] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6465.805507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6470.338988] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6470.540991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6475.751023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.751440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.762208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6475.766964] LustreError: Skipped 1 previous similar message [ 6475.778188] Lustre: Skipped 1 previous similar message [ 6478.791980] Lustre: server umount lustre-OST0000 complete [ 6488.757393] Lustre: server umount lustre-MDT0000 complete [ 6491.109992] LustreError: 171426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.138242] LustreError: 171426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6493.039249] LustreError: 172231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626664 with bad export cookie 9318701599451951685 [ 6493.044725] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6493.062612] LustreError: 172231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6493.380622] Lustre: server umount lustre-MDT0001 complete [ 6502.923816] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:44:31 (1788626671) [ 6509.737386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.420776] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6524.543477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6529.261230] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6539.852431] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6546.166545] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6552.479560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6559.137881] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6566.247520] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6579.224410] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:45:48 (1788626748) [ 6584.808956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.811244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6584.835246] Lustre: Skipped 4 previous similar messages [ 6591.009212] Lustre: server umount lustre-OST0000 complete [ 6596.598336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6602.608962] Lustre: server umount lustre-MDT0000 complete [ 6606.973677] LustreError: 174624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626778 with bad export cookie 9318701599451952567 [ 6606.975786] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.985134] LustreError: 174624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6607.378553] Lustre: server umount lustre-MDT0001 complete [ 6619.046623] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 6621.949684] Key type lgssc unregistered [ 6622.240556] LNet: 178372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6622.247418] LNetError: 178372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6622.266330] LNet: Removed LNI 192.168.206.101@tcp [ 6623.030232] Key type .llcrypt unregistered [ 6623.032188] Key type ._llcrypt unregistered [ 6641.581294] Key type ._llcrypt registered [ 6641.582987] Key type .llcrypt registered [ 6641.691591] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 6642.783049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6642.821865] alg: No test for adler32 (adler32-zlib) [ 6643.788610] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 6644.003418] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 6645.655322] Key type lgssc registered [ 6646.713718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6653.496218] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6653.530417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6669.088526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6674.207845] LustreError: 179613:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.101@tcp: failed processing log, type 4: rc = -110 [ 6679.328945] LustreError: 179613:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6679.351904] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6679.364050] LustreError: MGC192.168.206.101@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6679.383803] LustreError: 179613:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6679.394263] LustreError: 179613:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 6679.407545] LustreError: 179613:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6679.459425] Lustre: server umount lustre-OST0000 complete [ 6679.462885] LustreError: 179613:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6687.529422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6688.876994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6692.718663] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6698.467313] LustreError: 180112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6698.487963] LustreError: 180112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6700.085379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6700.357300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6704.275933] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6711.500971] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6716.196726] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6723.326646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.512697] LustreError: 181929:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6723.525296] Lustre: 181929:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.101@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6723.677354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6729.453395] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6736.766720] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6739.426942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.432528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6739.448652] Lustre: Skipped 2 previous similar messages [ 6739.458103] Lustre: Skipped 1 previous similar message [ 6744.546832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6744.553649] Lustre: Skipped 3 previous similar messages [ 6744.709881] Lustre: server umount lustre-MDT0000 complete [ 6748.051822] LustreError: 180093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626919 with bad export cookie 5637339860417491108 [ 6748.054342] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6748.071659] LustreError: 180093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6748.466766] Lustre: server umount lustre-MDT0001 complete [ 6763.098122] Lustre: server umount lustre-OST0000 complete [ 6771.154749] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 6780.615995] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 6813.241673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.425574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6813.451742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6813.511403] Lustre: lustre-MDT0000: new disk, initializing [ 6813.606781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6813.633668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6817.856478] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6828.853674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.002867] Lustre: 185451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6829.123243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6829.133712] Lustre: Skipped 1 previous similar message [ 6829.313521] Lustre: lustre-MDT0001: new disk, initializing [ 6829.522689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6829.584911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6829.621692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6834.881607] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6844.909681] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6850.093823] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6856.235031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.463795] Lustre: lustre-OST0000: new disk, initializing [ 6856.470219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6856.478129] Lustre: 187416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6856.557268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.815685] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6865.437585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6865.444949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6865.525123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6869.531236] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6873.308508] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6873.479076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6876.786437] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6877.007977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6880.747460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6880.755053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.755975] LustreError: Skipped 1 previous similar message [ 6880.764938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6880.813966] Lustre: Skipped 1 previous similar message [ 6884.671891] Lustre: server umount lustre-OST0000 complete [ 6890.984885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.985188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6890.988925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.988934] Lustre: Skipped 1 previous similar message [ 6891.009504] Lustre: Skipped 1 previous similar message [ 6896.096822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6896.102114] Lustre: Skipped 2 previous similar messages [ 6902.239129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6902.427343] Lustre: server umount lustre-MDT0000 complete [ 6905.313107] LustreError: 185462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6905.327284] LustreError: 185462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6905.528203] LustreError: 185442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627077 with bad export cookie 5637339860417491829 [ 6905.533360] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6905.541641] LustreError: 185442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6905.786975] Lustre: server umount lustre-MDT0001 complete [ 6912.391373] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:51:21 (1788627081) [ 6925.700632] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 6928.037550] Key type lgssc unregistered [ 6928.315631] LNet: 190358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6928.328217] LNetError: 190358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6928.355859] LNet: Removed LNI 192.168.206.101@tcp [ 6929.072417] Key type .llcrypt unregistered [ 6929.074467] Key type ._llcrypt unregistered [ 6946.911354] Key type ._llcrypt registered [ 6946.912908] Key type .llcrypt registered [ 6947.138869] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 6948.733685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6948.793039] alg: No test for adler32 (adler32-zlib) [ 6949.851748] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 6950.062208] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 6951.743174] Key type lgssc registered [ 6952.977968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6961.137861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6961.176846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.644057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.999263] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6972.899380] LustreError: 191572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.923260] LustreError: 191572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6973.111599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6973.464619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6977.441829] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 6984.034308] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6989.564880] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6996.178527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.419516] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6996.422571] LustreError: 193437:0:(llog.c:1696:llog_backup()) MGC192.168.206.101@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6996.428212] Lustre: 193437:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.101@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6996.518388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7001.952513] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7008.315945] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7012.833146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7012.836897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7012.848691] Lustre: Skipped 2 previous similar messages [ 7012.865171] Lustre: Skipped 2 previous similar messages [ 7016.467763] Lustre: server umount lustre-MDT0000 complete [ 7017.956529] LustreError: 191572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7017.975924] LustreError: 191572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7019.299400] LustreError: 191553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627191 with bad export cookie 17483427661662484464 [ 7019.312972] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7019.315569] LustreError: 191553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7019.586477] Lustre: server umount lustre-MDT0001 complete [ 7033.086132] Lustre: server umount lustre-OST0000 complete [ 7041.688147] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 7051.430159] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 7085.914039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.125937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7086.181272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7086.289312] Lustre: lustre-MDT0000: new disk, initializing [ 7086.451696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7086.499324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7090.703643] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7099.884671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.993683] Lustre: 196961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7100.049952] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7100.053973] Lustre: Skipped 1 previous similar message [ 7100.057572] LustreError: 197744:0:(llog.c:1696:llog_backup()) MGC192.168.206.101@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7100.071913] LustreError: 197744:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7100.077147] Lustre: 197744:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.101@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7100.103097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7100.105281] Lustre: Skipped 1 previous similar message [ 7100.185454] Lustre: lustre-MDT0001: new disk, initializing [ 7100.269535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7100.299709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7100.307265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7103.708473] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7112.437516] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7118.544389] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7124.823958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7125.012887] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7125.016436] Lustre: Skipped 1 previous similar message [ 7125.019629] LustreError: 198909:0:(llog.c:1696:llog_backup()) MGC192.168.206.101@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7125.025369] LustreError: 198909:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7125.030253] Lustre: 198909:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.101@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7125.036570] Lustre: 198909:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7125.053579] Lustre: lustre-OST0000: new disk, initializing [ 7125.056809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7125.061813] Lustre: 198932:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7125.114414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7130.619169] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7130.718942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7130.729257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7130.818830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7139.540777] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7143.440021] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7143.642917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7147.815764] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7148.053495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7151.587735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7151.593048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.598463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7151.644453] Lustre: Skipped 1 previous similar message [ 7156.708130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7156.715850] Lustre: Skipped 2 previous similar messages [ 7161.829740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7161.851083] Lustre: Skipped 1 previous similar message [ 7164.383237] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7164.695545] Lustre: server umount lustre-OST0000 complete [ 7172.068230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7172.082196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.083771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7172.121722] Lustre: Skipped 1 previous similar message [ 7182.306504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7182.318611] Lustre: Skipped 4 previous similar messages [ 7183.329392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7183.578160] Lustre: server umount lustre-MDT0000 complete [ 7187.034072] LustreError: 197756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627358 with bad export cookie 17483427661662485185 [ 7187.045524] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7187.049170] LustreError: 197756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7187.343524] Lustre: server umount lustre-MDT0001 complete [ 7195.062920] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:56:04 (1788627364) [ 7211.906699] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 7221.368403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.734837] LustreError: 202059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.773456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7226.230601] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7226.855623] LustreError: 202060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7231.969556] LustreError: 202059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7234.520095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.952859] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7241.421595] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7246.397191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7251.350883] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7258.243416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7258.425398] Lustre: lustre-OST0001: new disk, initializing [ 7258.428960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7258.435057] Lustre: 204365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7258.518431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7258.522921] Lustre: Skipped 2 previous similar messages [ 7263.430506] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7268.409513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7268.428848] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7268.526570] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7272.506867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7282.202841] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7283.798679] Lustre: DEBUG MARKER: ADD OST3 [ 7288.225229] loop: module loaded [ 7288.304380] loop0: detected capacity change from 0 to 204800000 [ 7288.586238] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7288.611672] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7288.636694] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7294.142683] loop0: detected capacity change from 0 to 204800000 [ 7295.294629] Lustre: DEBUG MARKER: STOP OST3 [ 7299.176169] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7299.380518] Lustre: lustre-OST0003: new disk, initializing [ 7299.385324] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7299.390935] Lustre: 206345:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7301.482248] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7301.687663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7301.697909] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7301.761524] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7303.475926] Lustre: Failing over lustre-OST0003 [ 7303.519276] LustreError: 206553:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7303.641461] Lustre: server umount lustre-OST0003 complete [ 7305.901514] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7306.446353] LustreError: 204889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.727098] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7306.742031] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.752711] Lustre: Skipped 1 previous similar message [ 7313.611666] Lustre: DEBUG MARKER: START OST3 again [ 7321.394976] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.695800] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7321.778375] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7323.013406] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7323.014309] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 7328.648343] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7352.170350] Lustre: Failing over lustre-OST0003 [ 7352.300235] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7352.320820] Lustre: Skipped 1 previous similar message [ 7352.329597] LustreError: 204889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7352.347358] LustreError: 204889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7352.439763] Lustre: server umount lustre-OST0003 complete [ 7372.772299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.783190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7372.797137] Lustre: Skipped 1 previous similar message [ 7372.818439] Lustre: Skipped 3 previous similar messages [ 7377.989791] Lustre: server umount lustre-MDT0000 complete [ 7382.199855] LustreError: 202040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627554 with bad export cookie 17483427661662486067 [ 7382.221061] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7387.041169] LustreError: 205802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7387.076075] LustreError: 205802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 7388.830422] Lustre: server umount lustre-MDT0001 complete [ 7392.899894] Lustre: server umount lustre-OST0000 complete [ 7396.989367] Lustre: server umount lustre-OST0001 complete [ 7404.870620] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 7417.125165] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 7454.394361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.626763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7454.646863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7454.750457] Lustre: lustre-MDT0000: new disk, initializing [ 7454.847361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7454.853247] Lustre: Skipped 2 previous similar messages [ 7454.871829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7459.648516] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7469.691391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.816870] Lustre: 211831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7470.070640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7474.135851] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7483.793239] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7488.632448] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7495.283655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.611979] Lustre: lustre-OST0000: new disk, initializing [ 7495.622388] Lustre: Skipped 1 previous similar message [ 7495.634101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7495.640819] Lustre: Skipped 2 previous similar messages [ 7495.660245] Lustre: 213812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7497.241218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7497.248825] Lustre: Skipped 1 previous similar message [ 7497.254667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7497.367026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7502.159378] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7510.008915] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7514.070762] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7514.275850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7518.368218] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7518.575577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7523.310245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7523.310613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.320397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7523.320405] Lustre: Skipped 4 previous similar messages [ 7523.323682] LustreError: Skipped 1 previous similar message [ 7523.353966] Lustre: Skipped 5 previous similar messages [ 7526.555122] Lustre: server umount lustre-OST0000 complete [ 7531.489687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7536.671982] Lustre: server umount lustre-MDT0000 complete [ 7538.656480] LustreError: 211842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7538.668898] LustreError: 211842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7539.833080] LustreError: 211825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627711 with bad export cookie 17483427661662488363 [ 7539.839823] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7539.844769] LustreError: 211825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7540.128563] Lustre: server umount lustre-MDT0001 complete [ 7543.484691] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:01:52 (1788627712) [ 7558.561531] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 7567.754659] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 7600.778647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.044414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7601.081751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7601.160405] Lustre: lustre-MDT0000: new disk, initializing [ 7601.227720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7601.234728] Lustre: Skipped 2 previous similar messages [ 7601.251882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7606.129943] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7617.318820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.490228] Lustre: 218423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7617.904846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7621.922237] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7630.230758] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7635.007391] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7641.082667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7641.424805] Lustre: 220394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7642.978725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7648.014980] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7657.201264] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7661.513898] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7661.721490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7666.372712] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7666.587744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7668.706641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7668.716057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.718054] LustreError: Skipped 1 previous similar message [ 7668.728938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.740735] Lustre: Skipped 3 previous similar messages [ 7668.761663] Lustre: Skipped 5 previous similar messages [ 7674.214518] Lustre: server umount lustre-OST0000 complete [ 7683.437223] Lustre: server umount lustre-MDT0000 complete [ 7684.064331] LustreError: 218433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7684.077845] LustreError: 218433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7686.369084] LustreError: 218414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627858 with bad export cookie 17483427661662489245 [ 7686.378353] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7686.382110] LustreError: 218414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7686.680191] Lustre: server umount lustre-MDT0001 complete [ 7694.735822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7698.907661] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7706.161277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7710.111367] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7716.300974] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7722.273495] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7728.966326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7735.672882] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7744.107558] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7756.258678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7756.269278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.287107] Lustre: Skipped 2 previous similar messages [ 7760.254619] Lustre: server umount lustre-OST0000 complete [ 7770.006064] Lustre: server umount lustre-MDT0000 complete [ 7773.683490] LustreError: 222590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627945 with bad export cookie 17483427661662490127 [ 7773.690707] LustreError: 222590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7773.691237] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7774.110598] Lustre: server umount lustre-MDT0001 complete [ 7884.264630] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 7886.673182] Key type lgssc unregistered [ 7886.947391] LNet: 226272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7886.952316] LNetError: 226272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7886.974093] LNet: Removed LNI 192.168.206.101@tcp [ 7887.657166] Key type .llcrypt unregistered [ 7887.664499] Key type ._llcrypt unregistered [ 7910.266402] Key type ._llcrypt registered [ 7910.268641] Key type .llcrypt registered [ 7910.389786] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 7924.754866] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 7925.748424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7925.797918] alg: No test for adler32 (adler32-zlib) [ 7926.896705] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 7927.173615] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 7928.903152] Key type lgssc registered [ 7930.724962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7964.453524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7964.482916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.695111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7965.757844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7965.876511] Lustre: lustre-MDT0000: new disk, initializing [ 7965.950569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7965.976351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7970.797852] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7980.424478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.505114] Lustre: 229619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7980.551282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7980.558777] Lustre: Skipped 1 previous similar message [ 7980.627142] Lustre: lustre-MDT0001: new disk, initializing [ 7980.711134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7980.745960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7980.760166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7984.962609] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 7993.626370] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7999.922166] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8006.934436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.256762] Lustre: lustre-OST0000: new disk, initializing [ 8007.261532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8007.274776] Lustre: 231589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8007.388613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.632395] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8014.423308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8014.438872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8014.553700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8022.273510] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8026.800714] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8027.028657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8031.821792] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8032.111782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8035.296110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8035.308105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8035.324957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8035.334243] Lustre: Skipped 1 previous similar message [ 8040.339165] Lustre: server umount lustre-OST0000 complete [ 8045.539794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.549257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.566084] Lustre: Skipped 2 previous similar messages [ 8050.657586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8050.666148] Lustre: Skipped 2 previous similar messages [ 8055.778532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.791100] Lustre: Skipped 1 previous similar message [ 8058.335120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8058.499716] Lustre: server umount lustre-MDT0000 complete [ 8062.173036] LustreError: 229613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628234 with bad export cookie 17499021328675713571 [ 8062.181908] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8062.184551] LustreError: 229613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8062.895592] Lustre: server umount lustre-MDT0001 complete [ 8083.072932] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 8095.414613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8096.096258] LustreError: 234525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.154495] LustreError: 234525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8096.254659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8101.177154] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8101.347074] LustreError: 234526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.469593] LustreError: 234525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.264417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.620203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8115.330919] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8117.853212] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8124.366131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8124.730085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.826214] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8138.826990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8138.977463] Lustre: lustre-OST0001: new disk, initializing [ 8138.980369] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8138.985616] Lustre: 236831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8145.132858] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8146.043480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8146.054926] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8146.203063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8154.466586] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8160.398755] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8172.569193] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:12:21 (1788628341) [ 8177.120348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8177.125918] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.131375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8177.134423] LustreError: Skipped 1 previous similar message [ 8177.184873] Lustre: Skipped 2 previous similar messages [ 8180.847377] Lustre: server umount lustre-MDT0000 complete [ 8182.253487] LustreError: 234520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.268640] LustreError: 234520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8184.572484] LustreError: 234506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628356 with bad export cookie 17499021328675714453 [ 8184.579916] LustreError: 234506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8184.580547] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8184.927836] Lustre: server umount lustre-MDT0001 complete [ 8198.944173] Lustre: server umount lustre-OST0000 complete [ 8213.273885] Lustre: server umount lustre-OST0001 complete [ 8220.499893] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 8233.510706] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 8269.698550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.939755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8269.965182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8270.024552] Lustre: lustre-MDT0000: new disk, initializing [ 8270.077530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.081484] Lustre: Skipped 1 previous similar message [ 8270.101431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8275.495472] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8289.027053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.095939] Lustre: 240778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8289.137517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8289.141030] Lustre: Skipped 1 previous similar message [ 8289.195765] Lustre: lustre-MDT0001: new disk, initializing [ 8289.286091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8289.304735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8293.661172] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8302.795484] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8308.302191] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8315.158224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8315.410568] Lustre: lustre-OST0000: new disk, initializing [ 8315.413631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8315.418415] Lustre: 242748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8315.510367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8315.517055] Lustre: Skipped 1 previous similar message [ 8316.622392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8316.632921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8316.800644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8321.136554] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8329.820598] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8334.204894] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8334.410521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8338.219321] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8338.360914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8342.502659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8342.505108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.512827] LustreError: Skipped 1 previous similar message [ 8342.513793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8342.513798] Lustre: Skipped 3 previous similar messages [ 8342.545534] Lustre: Skipped 2 previous similar messages [ 8346.584894] Lustre: server umount lustre-OST0000 complete [ 8350.688760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8350.698465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.708849] Lustre: server umount lustre-MDT0000 complete [ 8357.861700] LustreError: 240791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.882409] LustreError: 240791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8360.144430] LustreError: 240772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628532 with bad export cookie 17499021328675716588 [ 8360.153357] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8360.154411] LustreError: 240772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8360.554303] Lustre: server umount lustre-MDT0001 complete [ 8368.478910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8368.995038] LustreError: 244962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.456992] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8382.239971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.603318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.608918] Lustre: Skipped 1 previous similar message [ 8387.104296] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8394.544969] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8399.996718] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8406.593994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8413.186115] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8420.015233] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8566.754674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8566.775776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8566.796296] Lustre: Skipped 1 previous similar message [ 8566.821712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8566.832624] Lustre: Skipped 4 previous similar messages [ 8570.155145] Lustre: server umount lustre-OST0000 complete [ 8576.995648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8577.001221] LustreError: Skipped 1 previous similar message [ 8588.255134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8588.427058] Lustre: server umount lustre-MDT0000 complete [ 8592.218022] LustreError: 244942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628764 with bad export cookie 17499021328675717470 [ 8592.218815] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8592.229557] LustreError: 244942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8592.647751] Lustre: server umount lustre-MDT0001 complete [ 8685.620530] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 8687.915790] Key type lgssc unregistered [ 8688.172552] LNet: 248572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8688.180962] LNetError: 248572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8688.191725] LNet: Removed LNI 192.168.206.101@tcp [ 8688.913175] Key type .llcrypt unregistered [ 8688.921126] Key type ._llcrypt unregistered [ 8708.812890] Key type ._llcrypt registered [ 8708.814527] Key type .llcrypt registered [ 8708.899378] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 8722.233655] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 8723.138787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8723.150449] alg: No test for adler32 (adler32-zlib) [ 8724.205950] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 8724.426645] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 8726.103138] Key type lgssc registered [ 8727.114312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8766.651235] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 8777.308092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8777.348899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.586369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8778.622974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8778.718674] Lustre: lustre-MDT0000: new disk, initializing [ 8778.798529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8778.821085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8783.091447] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8793.740446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8793.849595] Lustre: 252637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8793.906788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8793.909647] Lustre: Skipped 1 previous similar message [ 8793.957734] Lustre: lustre-MDT0001: new disk, initializing [ 8794.025046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8794.052805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8794.063709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8798.280909] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8802.529797] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8809.721538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8809.929212] Lustre: lustre-OST0000: new disk, initializing [ 8809.933812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8809.946361] Lustre: 254269:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8810.036126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8815.735755] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8816.149869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8816.160084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8816.250635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8825.355627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8825.521504] Lustre: lustre-OST0001: new disk, initializing [ 8825.526518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8825.533064] Lustre: 255139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8825.617667] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8830.773355] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8833.634875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8833.648665] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8833.771945] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8840.855681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8850.829686] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8859.621366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8859.633667] Lustre: Skipped 2 previous similar messages [ 8859.641372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8859.648572] Lustre: Skipped 3 previous similar messages [ 8864.743651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8864.751587] Lustre: Skipped 1 previous similar message [ 8864.941340] Lustre: server umount lustre-MDT0000 complete [ 8867.875471] LustreError: 252627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629039 with bad export cookie 16985068466571507366 [ 8867.879558] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8867.888636] LustreError: 252627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8868.288561] Lustre: server umount lustre-MDT0001 complete [ 8881.694297] Lustre: server umount lustre-OST0000 complete [ 8894.889142] Lustre: server umount lustre-OST0001 complete [ 8906.296367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8913.158347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8922.076485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8934.520143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8953.744614] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 8963.402663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8963.422158] Lustre: lustre-MDT0000: reset Object Index mappings [ 8963.852130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.407103] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8969.185378] LustreError: 260149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8969.199880] LustreError: 260149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8973.289390] LustreError: 260150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8974.400752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8974.420082] Lustre: lustre-MDT0001: reset Object Index mappings [ 8974.735306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8978.164128] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8980.587122] Lustre: 261290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8986.925631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8988.340898] LustreError: 261644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8988.365220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8992.842770] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 8993.766262] LustreError: 261737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8998.884148] LustreError: 261643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9000.620826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.868515] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9000.875838] Lustre: Skipped 1 previous similar message [ 9001.916698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9006.550863] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9013.835408] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9017.288589] Lustre: 263162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9022.434704] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9022.438880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9022.451158] Lustre: Skipped 3 previous similar messages [ 9022.462359] Lustre: Skipped 3 previous similar messages [ 9024.125165] Lustre: server umount lustre-MDT0000 complete [ 9027.016887] LustreError: 260129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629198 with bad export cookie 16985068466571514149 [ 9027.029445] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.029955] LustreError: 260129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9027.368678] Lustre: server umount lustre-MDT0001 complete [ 9040.856298] Lustre: server umount lustre-OST0000 complete [ 9043.937129] Lustre: 250143:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788629199/real 1788629199] req@ffff9d56886ad500 x1875513333372928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788629215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9043.968203] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.981819] Lustre: Skipped 1 previous similar message [ 9044.626959] Lustre: server umount lustre-OST0001 complete [ 9061.899577] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:27:11 (1788629231) [ 9075.676743] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 9085.111342] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 9114.827662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.001816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9115.023616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9115.068198] Lustre: lustre-MDT0000: new disk, initializing [ 9115.126563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9115.147536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9118.043629] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9126.468524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9126.568310] Lustre: 267412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9126.626443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9126.630165] Lustre: Skipped 1 previous similar message [ 9126.698249] Lustre: lustre-MDT0001: new disk, initializing [ 9126.790428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9126.802158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9130.133161] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9137.180996] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9141.442140] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9146.596025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9146.850923] Lustre: lustre-OST0000: new disk, initializing [ 9146.854775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9146.860115] Lustre: 269378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9148.024780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9148.031994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9148.099577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9151.890368] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9160.314991] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9165.105101] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9165.294373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9169.326453] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9169.532814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9173.992763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9173.994255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.001409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9174.036862] Lustre: Skipped 1 previous similar message [ 9177.392970] Lustre: server umount lustre-OST0000 complete [ 9183.200893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9183.206426] LustreError: Skipped 1 previous similar message [ 9183.211591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.222804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9183.232054] Lustre: Skipped 1 previous similar message [ 9186.495478] Lustre: server umount lustre-MDT0000 complete [ 9189.346610] LustreError: 267417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.371511] LustreError: 267417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9189.631578] LustreError: 267404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629361 with bad export cookie 16985068466571516249 [ 9189.637650] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9189.643433] LustreError: 267404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9189.884951] Lustre: server umount lustre-MDT0001 complete [ 9206.160921] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 9214.444091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9214.735646] LustreError: 272313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.766386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9214.774056] Lustre: Skipped 2 previous similar messages [ 9218.556648] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9225.369958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.301877] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9231.555127] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9236.371029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9241.648529] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9248.124338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9248.250295] Lustre: lustre-OST0001: new disk, initializing [ 9248.252987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9248.258411] Lustre: 274620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9249.643369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9249.651675] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9249.764121] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9253.022350] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9260.294688] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9263.013323] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9267.602093] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9267.644189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9274.929135] Lustre: server umount lustre-MDT0000 complete [ 9276.390162] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9276.398406] LustreError: 272308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9276.406809] Lustre: Skipped 3 previous similar messages [ 9276.424282] LustreError: 272308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9278.422360] LustreError: 272295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629450 with bad export cookie 16985068466571517131 [ 9278.425497] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9278.434802] LustreError: 272295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9278.699674] Lustre: server umount lustre-MDT0001 complete [ 9291.898958] Lustre: server umount lustre-OST0000 complete [ 9304.488933] Lustre: server umount lustre-OST0001 complete [ 9319.533781] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 9327.331984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.059214] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9337.268435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.249888] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9342.008864] Lustre: 278810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9347.031981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9347.358274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9347.363468] Lustre: Skipped 5 previous similar messages [ 9348.387508] LustreError: 279166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9348.399884] LustreError: 279166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9352.503464] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9353.535894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9360.284733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9365.555583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9365.842544] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9372.683984] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9375.587291] Lustre: 280684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9380.187777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9380.268480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9389.401475] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:32:38 (1788629558) [ 9404.384503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9404.391627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.404913] Lustre: Skipped 1 previous similar message [ 9404.414895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.424896] Lustre: Skipped 2 previous similar messages [ 9410.089225] Lustre: server umount lustre-MDT0000 complete [ 9413.532842] LustreError: 277651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629585 with bad export cookie 16985068466571523438 [ 9413.542653] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9413.543249] LustreError: 277651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9419.939131] Lustre: server umount lustre-MDT0001 complete [ 9429.531023] Lustre: server umount lustre-OST0000 complete [ 9446.879182] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9447.020866] Lustre: server umount lustre-OST0001 complete [ 9453.265615] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [ 9461.417829] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 9502.200697] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 9512.006876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9512.264246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9512.307920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9512.434309] Lustre: lustre-MDT0000: new disk, initializing [ 9512.568307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9517.024724] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9526.799376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9526.912455] Lustre: 284928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9527.206809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9531.631924] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9542.807927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.176715] Lustre: 286558:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9544.960033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9549.171454] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9558.727126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9558.941244] Lustre: 287428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9560.803698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9560.814131] Lustre: Skipped 1 previous similar message [ 9560.873451] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9564.544701] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9573.098607] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9576.579812] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9576.585517] Lustre: Skipped 1 previous similar message [ 9590.834691] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:35:59 (1788629759) [ 9593.829233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9593.843829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9593.856167] Lustre: Skipped 5 previous similar messages [ 9593.860193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9593.866904] Lustre: Skipped 8 previous similar messages [ 9598.923216] Lustre: server umount lustre-MDT0000 complete [ 9601.983429] LustreError: 285732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629773 with bad export cookie 16985068466571531327 [ 9601.988758] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9602.002364] LustreError: 285732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9602.023789] LustreError: 284935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9602.034579] LustreError: 284935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 9602.313908] Lustre: server umount lustre-MDT0001 complete [ 9608.345095] Lustre: server umount lustre-OST0000 complete [ 9612.316083] Lustre: server umount lustre-OST0001 complete [ 9620.400724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9620.875031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9620.879912] Lustre: Skipped 5 previous similar messages [ 9625.018194] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9632.802546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9637.122970] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9643.771327] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9649.066394] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9656.482662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9662.456410] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9669.451791] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9684.376331] loop1: detected capacity change from 0 to 204800000 [ 9684.661395] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9684.684613] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9684.696953] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9690.021451] loop1: detected capacity change from 0 to 204800000 [ 9695.389773] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9695.493082] LustreError: 289662:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs15657: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9695.515781] LustreError: 289662:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs15657 [ 9695.523359] LustreError: 289662:0:(mgs_handler.c:517:mgs_target_reg()) fs15657-MDT0001: Error checking target index: rc = -13 [ 9695.530532] LustreError: fs15657-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9695.536618] LustreError: 293391:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [ 9695.541580] LustreError: 293391:0:(tgt_mount.c:2132:server_put_super()) no obd fs15657-MDT0001 [ 9695.546894] LustreError: 293391:0:(tgt_mount.c:116:server_deregister_mount()) fs15657-MDT0001 not registered [ 9695.571174] Lustre: server umount fs15657-MDT0001 complete [ 9695.573625] LustreError: 293391:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9703.283359] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9703.411712] Lustre: Setting parameter fs15657-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15657-MDT0001 [ 9703.465238] Lustre: srv-fs15657-MDT0001: No data found on store. Initialize space. [ 9703.470086] Lustre: Skipped 4 previous similar messages [ 9703.563393] Lustre: fs15657-MDT0001: new disk, initializing [ 9703.567569] Lustre: Skipped 3 previous similar messages [ 9703.662346] Lustre: srv-fs15657-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9703.671153] Lustre: 293857:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15657-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9703.676924] LustreError: 293857:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15657-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9707.205717] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9713.690558] Lustre: server umount fs15657-MDT0001 complete [ 9726.159858] loop1: detected capacity change from 0 to 204800000 [ 9730.701696] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.957887] Lustre: 295427:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15657-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9730.967965] LustreError: 295427:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15657-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9734.478091] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9738.498656] Lustre: server umount fs15657-MDT0001 complete [ 9742.829961] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9747.137950] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.1@tcp (stopping) [ 9747.150205] Lustre: Skipped 6 previous similar messages [ 9747.939386] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.953850] Lustre: Skipped 6 previous similar messages [ 9761.247293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9761.516890] Lustre: server umount lustre-MDT0000 complete [ 9769.706805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.828227] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9775.391625] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9775.618740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9782.575516] loop1: detected capacity change from 0 to 204800000 [ 9787.864614] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.332219] Lustre: 297875:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15657-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9788.344108] LustreError: 297875:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15657-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9792.439360] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9797.259191] Lustre: server umount fs15657-MDT0001 complete [ 9816.034696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9816.042829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9816.048300] Lustre: Skipped 11 previous similar messages [ 9829.344666] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9829.510057] Lustre: server umount lustre-OST0000 complete [ 9837.033742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9847.265046] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9847.580087] Lustre: server umount lustre-MDT0000 complete [ 9851.008541] LustreError: 289655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630022 with bad export cookie 16985068466571534008 [ 9851.016817] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9851.033028] LustreError: 289655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9851.445033] Lustre: server umount lustre-MDT0001 complete [ 9861.217349] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [ 9863.885773] Key type lgssc unregistered [ 9864.143759] LNet: 300221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9864.155392] LNetError: 300221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9864.173682] LNet: Removed LNI 192.168.206.101@tcp [ 9865.162297] Key type .llcrypt unregistered [ 9865.167331] Key type ._llcrypt unregistered [ 9882.575058] Key type ._llcrypt registered [ 9882.578452] Key type .llcrypt registered [ 9882.713372] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [ 9883.480474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9883.557128] alg: No test for adler32 (adler32-zlib) [ 9884.623523] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [ 9884.860041] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [ 9886.527183] Key type lgssc registered [ 9887.707268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9895.085675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9895.121361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.563111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.560519] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9906.159616] LustreError: 301488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9906.174486] LustreError: 301488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9910.285701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9910.698358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9914.845364] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9921.520621] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9926.871424] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9933.206107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9933.568614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9935.590776] LustreError: 303459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9939.177433] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9940.964755] LustreError: 303459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9945.617942] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9947.107828] LustreError: 303319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.145052] LustreError: 303319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9952.229084] LustreError: 303319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.247184] LustreError: 303319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9961.010220] loop1: detected capacity change from 0 to 204800000 [ 9961.162128] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9961.181087] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9961.197853] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9962.484936] LustreError: 303319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9962.503533] LustreError: 303319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9965.250808] loop1: detected capacity change from 0 to 204800000 [ 9969.805988] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9969.933549] Lustre: fs31052-OST0000: new disk, initializing [ 9969.940281] Lustre: srv-fs31052-OST0000: No data found on store. Initialize space. [ 9969.960450] Lustre: 305224:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9970.038268] Lustre: fs31052-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9975.372938] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [ 9979.051365] LustreError: 303316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.062016] LustreError: 303316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9983.474951] Lustre: server umount fs31052-OST0000 complete [ 9995.865484] loop1: detected capacity change from 0 to 204800000 [10001.252426] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10001.557682] Lustre: fs31052-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10007.357648] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10013.669053] LustreError: 303459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.669566] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10013.699977] LustreError: 303459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [10015.430113] loop2: detected capacity change from 0 to 204800000 [10015.581233] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10015.592568] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10015.609159] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10018.792987] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10020.553398] loop2: detected capacity change from 0 to 204800000 [10026.522724] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10026.615269] LustreError: fs31052-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10026.627784] LustreError: fs31052-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10026.637859] LustreError: 308264:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10026.649342] LustreError: 308264:0:(tgt_mount.c:2132:server_put_super()) no obd fs31052-OST0001 [10026.654868] LustreError: 308264:0:(tgt_mount.c:116:server_deregister_mount()) fs31052-OST0001 not registered [10026.697358] Lustre: server umount fs31052-OST0001 complete [10026.700711] LustreError: 308264:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10035.698818] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10035.881945] Lustre: fs31052-OST0001: new disk, initializing [10035.891488] Lustre: srv-fs31052-OST0001: No data found on store. Initialize space. [10035.901521] Lustre: 308816:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10035.991249] Lustre: fs31052-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10041.281384] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10048.744516] Lustre: server umount fs31052-OST0000 complete [10056.893971] Lustre: server umount fs31052-OST0001 complete [10067.075332] loop1: detected capacity change from 0 to 204800000 [10072.199785] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10072.374559] Lustre: fs31052-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10078.254868] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10080.229794] LustreError: 303459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.256324] LustreError: 303459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10084.615921] loop2: detected capacity change from 0 to 204800000 [10090.210579] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10095.536138] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10105.237640] Lustre: server umount fs31052-OST0000 complete [10114.684507] Lustre: server umount fs31052-OST0001 complete [10125.282673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10125.291547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.312496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10126.323990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10126.328853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.637639] Lustre: server umount lustre-OST0000 complete [10136.033648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10136.039467] LustreError: Skipped 1 previous similar message [10136.043255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10136.064428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10136.073709] Lustre: Skipped 1 previous similar message [10141.666843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10141.675847] Lustre: Skipped 2 previous similar messages [10146.784774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10146.797888] Lustre: Skipped 1 previous similar message [10149.343456] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10149.571668] Lustre: server umount lustre-MDT0000 complete [10152.867187] LustreError: 301468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630324 with bad export cookie 16516892499210852831 [10152.869893] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10152.878982] LustreError: 301468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10153.155526] Lustre: server umount lustre-MDT0001 complete [10162.767146] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [10165.007838] Key type lgssc unregistered [10165.280154] LNet: 314735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10165.289661] LNetError: 314735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10165.309079] LNet: Removed LNI 192.168.206.101@tcp [10166.095239] Key type .llcrypt unregistered [10166.098161] Key type ._llcrypt unregistered [10191.467615] Key type ._llcrypt registered [10191.470648] Key type .llcrypt registered [10191.568638] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [10203.433479] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [10204.461571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10204.504417] alg: No test for adler32 (adler32-zlib) [10205.501181] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [10205.726869] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [10207.383315] Key type lgssc registered [10208.257943] Lustre: Echo OBD driver; http://www.lustre.org/ [10238.600614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10238.626751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.895055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10239.943723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10240.054732] Lustre: lustre-MDT0000: new disk, initializing [10240.168504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10240.197332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10244.261668] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10252.866335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10252.997195] Lustre: 318568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10253.064242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10253.072886] Lustre: Skipped 1 previous similar message [10253.137156] Lustre: lustre-MDT0001: new disk, initializing [10253.224477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10253.262357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10253.274660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10257.752910] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10266.826917] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10272.328568] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10279.051501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10279.374509] Lustre: lustre-OST0000: new disk, initializing [10279.378705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10279.389379] Lustre: 320534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10279.473419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10285.775327] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10289.781243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10289.786534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10289.903513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10294.287240] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10296.265654] Lustre: Mounted lustre-client - version 2.17.57_85_gb449de9 [10297.216589] Lustre: Unmounted lustre-client [10305.505931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10305.526456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10305.551529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10305.566105] Lustre: Skipped 1 previous similar message [10306.538447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10306.544339] Lustre: Skipped 1 previous similar message [10311.650786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10311.657703] Lustre: Skipped 1 previous similar message [10316.779231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10316.795361] Lustre: Skipped 1 previous similar message [10319.327302] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10319.507620] Lustre: server umount lustre-OST0000 complete [10324.965448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10324.976447] LustreError: Skipped 1 previous similar message [10324.980943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.987960] Lustre: Skipped 1 previous similar message [10324.995073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10327.012539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10328.784717] Lustre: server umount lustre-MDT0000 complete [10332.090841] LustreError: 318561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630504 with bad export cookie 11591020471239034674 [10332.091264] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10332.100120] LustreError: 318561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10332.418975] Lustre: server umount lustre-MDT0001 complete [10341.782046] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [10344.307419] Key type lgssc unregistered [10348.064432] LNet: 1 local NIs in recovery (showing 1): 192.168.206.101@tcp [10348.639866] LNet: 322697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10348.652328] LNetError: 322697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10425.510299] LNet: Removed LNI 192.168.206.101@tcp [10426.117386] Key type .llcrypt unregistered [10426.122208] Key type ._llcrypt unregistered [10441.257164] Key type ._llcrypt registered [10441.262734] Key type .llcrypt registered [10441.387285] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [10456.166162] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [10457.250840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10457.291717] alg: No test for adler32 (adler32-zlib) [10458.243711] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [10458.416227] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [10460.071152] Key type lgssc registered [10461.186391] Lustre: Echo OBD driver; http://www.lustre.org/ [10485.875405] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:50:55 (1788630655) [10491.771569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10491.808750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10493.057364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10493.098346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10493.173473] Lustre: lustre-MDT0000: new disk, initializing [10493.238747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10493.252419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10496.823857] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10504.888230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.973349] Lustre: 325913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10505.024792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10505.028558] Lustre: Skipped 1 previous similar message [10505.087747] Lustre: lustre-MDT0001: new disk, initializing [10505.171563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10505.221320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10505.233369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10508.983555] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10516.793463] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10522.143290] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10528.960522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10529.289834] Lustre: lustre-OST0000: new disk, initializing [10529.299541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10529.313105] Lustre: 327879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10529.417713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10535.999651] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10537.559769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10537.571350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10537.676428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10544.652448] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10563.581457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10563.582232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10563.600102] LustreError: Skipped 1 previous similar message [10563.600801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10563.667177] Lustre: Skipped 1 previous similar message [10565.848910] Lustre: server umount lustre-OST0000 complete [10571.747628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10571.752863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.788282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10571.801507] Lustre: Skipped 1 previous similar message [10573.796119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10573.805718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10573.833778] Lustre: Skipped 1 previous similar message [10576.097677] Lustre: server umount lustre-MDT0000 complete [10579.672233] LustreError: 325904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630751 with bad export cookie 11987817170932329528 [10579.672498] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10579.686757] LustreError: 325904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10580.083290] Lustre: server umount lustre-MDT0001 complete [10588.899899] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [10591.444775] Key type lgssc unregistered [10591.790927] LNet: 329787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10591.824131] LNetError: 329787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10591.860485] LNet: Removed LNI 192.168.206.101@tcp [10592.948977] Key type .llcrypt unregistered [10592.953888] Key type ._llcrypt unregistered [10611.034379] Key type ._llcrypt registered [10611.039344] Key type .llcrypt registered [10611.159245] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [10611.903864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10611.988067] alg: No test for adler32 (adler32-zlib) [10613.087552] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [10613.384773] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [10615.055173] Key type lgssc registered [10616.379506] Lustre: Echo OBD driver; http://www.lustre.org/ [10624.136464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10624.150839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10625.673059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10629.536291] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10635.234520] LustreError: 331059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.247915] LustreError: 331059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10637.522828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10638.066530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10642.786235] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10649.823029] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10654.838710] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10661.183878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10661.549723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10667.341057] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10674.252157] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10686.080383] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10688.685846] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10694.625517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10694.637816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10694.664908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10696.674401] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10696.678632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10696.692065] Lustre: Skipped 2 previous similar messages [10699.907688] Lustre: server umount lustre-MDT0000 complete [10701.796790] LustreError: 331787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10701.811887] LustreError: 331787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10704.990099] LustreError: 331040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630876 with bad export cookie 8287073666357041178 [10705.003130] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10705.014436] LustreError: 331040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10706.913461] LustreError: 331054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.913619] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.913624] Lustre: Skipped 1 previous similar message [10706.915075] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10706.931286] LustreError: 331054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10706.947658] Lustre: Skipped 1 previous similar message [10709.920309] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10709.920310] LustreError: 331055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10709.920320] LustreError: 331055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10709.925360] Lustre: Skipped 1 previous similar message [10711.482803] Lustre: server umount lustre-MDT0001 complete [10714.886724] Lustre: server umount lustre-OST0000 complete [10722.584627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.260562] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10730.050189] Lustre: 335238:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.101@tcp [10732.314870] Lustre: server umount MGS complete [10741.748308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10742.243116] LustreError: 335809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.338577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10747.070966] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10752.492894] LustreError: 335810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10752.511607] LustreError: 335810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10756.483462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10756.875537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10761.751646] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10770.674941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10771.122082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10777.199338] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10783.781384] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10785.097725] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10790.715146] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10792.349416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10797.024935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10797.040309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10797.059377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10798.048484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10802.975724] Lustre: server umount lustre-OST0000 complete [10808.299067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10808.299995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10808.301447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10808.301786] Lustre: Skipped 2 previous similar messages [10808.313202] Lustre: Skipped 2 previous similar messages [10813.193977] Lustre: server umount lustre-MDT0000 complete [10813.413427] LustreError: 335810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.430195] LustreError: 335810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10816.970513] LustreError: 335790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630988 with bad export cookie 8287073666357042018 [10816.980862] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10816.983266] LustreError: 335790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10817.300854] Lustre: server umount lustre-MDT0001 complete [10828.985192] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [10832.307363] Key type lgssc unregistered [10832.809591] LNet: 339244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10832.818833] LNetError: 339244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10832.836682] LNet: Removed LNI 192.168.206.101@tcp [10833.898323] Key type .llcrypt unregistered [10833.903485] Key type ._llcrypt unregistered [10855.596684] Key type ._llcrypt registered [10855.599866] Key type .llcrypt registered [10855.703828] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [10874.581902] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [10876.373211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10876.432075] alg: No test for adler32 (adler32-zlib) [10877.514889] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [10877.780857] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [10879.559816] Key type lgssc registered [10880.485333] Lustre: Echo OBD driver; http://www.lustre.org/ [10917.656072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10917.685074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10919.032331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10919.096912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10919.237464] Lustre: lustre-MDT0000: new disk, initializing [10919.319439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10919.335407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10924.160767] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10934.718883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10934.857210] Lustre: 342405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10934.930465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10934.938573] Lustre: Skipped 1 previous similar message [10935.030414] Lustre: lustre-MDT0001: new disk, initializing [10935.126401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10935.196247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10935.202786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10939.753253] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10948.651806] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10954.471823] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10961.653856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10961.901489] Lustre: lustre-OST0000: new disk, initializing [10961.905726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10961.910916] Lustre: 344373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10962.003325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10968.103617] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [10970.716286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10970.725281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10970.861541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10977.818473] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10983.370969] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10983.621212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10989.591596] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10989.900556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10996.708993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10996.714817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10996.739487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10996.747459] Lustre: Skipped 1 previous similar message [10998.886923] Lustre: server umount lustre-OST0000 complete [11006.957363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11006.959047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11006.984863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11006.984878] Lustre: Skipped 1 previous similar message [11007.039453] Lustre: Skipped 1 previous similar message [11012.064864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11012.070522] Lustre: Skipped 1 previous similar message [11016.162540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11016.176146] Lustre: Skipped 1 previous similar message [11018.207137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11018.502857] Lustre: server umount lustre-MDT0000 complete [11023.182182] LustreError: 342397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631195 with bad export cookie 11558383445054280058 [11023.190682] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11023.193477] LustreError: 342397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11023.748221] Lustre: server umount lustre-MDT0001 complete [11033.007241] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:00:01 (1788631201) [11051.615597] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [11065.166542] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [11104.706344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11104.919594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11104.942131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11105.034804] Lustre: lustre-MDT0000: new disk, initializing [11105.128725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11105.148728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11109.992665] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [11121.265494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.399109] Lustre: 349144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11121.507940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11121.514238] Lustre: Skipped 1 previous similar message [11121.661050] Lustre: lustre-MDT0001: new disk, initializing [11121.803576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11121.864626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11121.893683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11127.096074] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [11136.787817] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11143.181656] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11150.102400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11150.333480] Lustre: lustre-OST0000: new disk, initializing [11150.336869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11150.344893] Lustre: 351113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11150.416331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11157.117292] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [11159.088046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11159.094541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11159.247072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11167.538517] Lustre: DEBUG MARKER: oleg601-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11205.617280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11205.620829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11205.639130] LustreError: Skipped 1 previous similar message [11205.640386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11205.664097] Lustre: Skipped 1 previous similar message [11205.686936] Lustre: Skipped 1 previous similar message [11209.996644] Lustre: server umount lustre-OST0000 complete [11214.342139] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11214.352802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11214.363642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11220.760916] Lustre: server umount lustre-MDT0000 complete [11220.963980] LustreError: 351933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11220.993600] LustreError: 351933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11224.703748] LustreError: 349137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631396 with bad export cookie 11558383445054280940 [11224.717519] LustreError: 349137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11224.720379] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11225.128645] Lustre: server umount lustre-MDT0001 complete [11236.119437] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11239.153761] Key type lgssc unregistered [11239.468129] LNet: 353289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11239.477699] LNetError: 353289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11239.492403] LNet: Removed LNI 192.168.206.101@tcp [11240.544641] Key type .llcrypt unregistered [11240.548749] Key type ._llcrypt unregistered [11329.022834] Key type ._llcrypt registered [11329.027040] Key type .llcrypt registered [11329.131606] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11329.539374] Key type .llcrypt unregistered [11329.544951] Key type ._llcrypt unregistered [11413.445620] Key type ._llcrypt registered [11413.447382] Key type .llcrypt registered [11413.551400] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11413.799733] Key type .llcrypt unregistered [11413.803516] Key type ._llcrypt unregistered [11441.828761] Key type ._llcrypt registered [11441.835870] Key type .llcrypt registered [11441.952583] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11442.215230] Key type .llcrypt unregistered [11442.218039] Key type ._llcrypt unregistered [11468.445560] Key type ._llcrypt registered [11468.458583] Key type .llcrypt registered [11468.569233] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11468.853437] Key type .llcrypt unregistered [11468.859973] Key type ._llcrypt unregistered [11541.220266] Key type ._llcrypt registered [11541.226370] Key type .llcrypt registered [11541.331916] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11541.591660] Key type .llcrypt unregistered [11541.596268] Key type ._llcrypt unregistered [11570.027230] Key type ._llcrypt registered [11570.030695] Key type .llcrypt registered [11570.162079] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11570.531342] Key type .llcrypt unregistered [11570.533398] Key type ._llcrypt unregistered [11595.503596] Key type ._llcrypt registered [11595.505796] Key type .llcrypt registered [11595.620822] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11595.979462] Key type .llcrypt unregistered [11595.984363] Key type ._llcrypt unregistered [11622.172860] Key type ._llcrypt registered [11622.184093] Key type .llcrypt registered [11622.315838] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11622.676341] Key type .llcrypt unregistered [11622.680243] Key type ._llcrypt unregistered [11648.697763] Key type ._llcrypt registered [11648.699500] Key type .llcrypt registered [11648.790773] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11649.024140] Key type .llcrypt unregistered [11649.032778] Key type ._llcrypt unregistered [11674.795899] Key type ._llcrypt registered [11674.798158] Key type .llcrypt registered [11674.867610] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11675.128802] Key type .llcrypt unregistered [11675.130919] Key type ._llcrypt unregistered [11725.295348] Key type ._llcrypt registered [11725.300873] Key type .llcrypt registered [11725.393812] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11725.610504] Key type .llcrypt unregistered [11725.613991] Key type ._llcrypt unregistered [11751.558743] Key type ._llcrypt registered [11751.567581] Key type .llcrypt registered [11751.714861] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11752.014296] Key type .llcrypt unregistered [11752.016378] Key type ._llcrypt unregistered [11781.241414] Key type ._llcrypt registered [11781.246826] Key type .llcrypt registered [11781.379571] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [11781.823577] Key type .llcrypt unregistered [11781.826277] Key type ._llcrypt unregistered [11828.461191] Key type ._llcrypt registered [11828.464047] Key type .llcrypt registered [11828.604345] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [11844.977649] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [11846.560838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11846.586037] alg: No test for adler32 (adler32-zlib) [11847.706976] Lustre: Lustre: Build Version: 2.17.57_85_gb449de9 [11848.036287] LNet: Added LNI 192.168.206.101@tcp [8/256/0/180] [11849.783152] Key type lgssc registered [11851.155626] Lustre: Echo OBD driver; http://www.lustre.org/ [11884.291417] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11884.323830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11885.466641] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11885.489902] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11885.581628] Lustre: tst250fs-MDT0000: new disk, initializing [11885.700419] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11885.721756] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11890.380712] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [11899.476132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.614178] Lustre: 373413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11899.660481] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11899.664650] Lustre: Skipped 1 previous similar message [11899.712150] Lustre: tst250fs-MDT0001: new disk, initializing [11899.770221] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11899.816491] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11899.825204] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11903.459283] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [11907.770891] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11913.469737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11913.694174] Lustre: tst250fs-OST0000: new disk, initializing [11913.697512] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11913.702864] Lustre: 375046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11913.794123] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11917.851345] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11917.868440] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11918.000070] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11919.704208] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [11928.453801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11928.608218] Lustre: tst250fs-OST0001: new disk, initializing [11928.613273] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11928.620870] Lustre: 375920:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11928.717109] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11933.169340] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_default_debug -1 all [11937.343742] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11937.351856] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11937.448293] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11937.737872] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11956.192415] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11956.198824] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.209857] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11958.248977] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11958.249585] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11958.260058] Lustre: Skipped 2 previous similar messages [11958.280747] Lustre: Skipped 2 previous similar messages [11961.533978] Lustre: server umount tst250fs-MDT0000 complete [11964.232601] LustreError: 375045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632136 with bad export cookie 8204591253241405096 [11964.236986] LustreError: MGC192.168.206.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11964.245787] LustreError: 375045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11964.474258] Lustre: server umount tst250fs-MDT0001 complete [11977.790781] Lustre: server umount tst250fs-OST0000 complete [11990.371195] Lustre: server umount tst250fs-OST0001 complete [12006.175746] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [12014.465287] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [12046.449542] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:16:55 (1788632215) [12047.633902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12061.699603] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing set_hostid [12070.212438] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing load_modules_local [12095.727826] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11802 sec ========= 14:17:45 (1788632265) [12097.084824] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:17:46 (1788632266) === [12099.645338] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:17:48 (1788632268) === [12129.803559] Lustre: DEBUG MARKER: oleg601-server.virtnet: executing unload_modules_local [12132.037061] Key type lgssc unregistered [12132.258784] LNet: 385057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12132.270429] LNetError: 385057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12132.292327] LNet: Removed LNI 192.168.206.101@tcp [12133.036776] Key type .llcrypt unregistered [12133.039320] Key type ._llcrypt unregistered