[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 451538502 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002205] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006849] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009102] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011039] SELinux: Initializing. [ 0.012048] *** VALIDATE selinux *** [ 0.020545] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025676] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.031509] *** VALIDATE proc *** [ 0.032257] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035185] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042037] debug: unmapping init [mem 0xffffffff90459000-0xffffffff90460fff] [ 0.044201] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045705] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053332] rcu: Hierarchical SRCU implementation. [ 0.055573] smp: Bringing up secondary CPUs ... [ 0.056617] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.060548] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130316] node 0 deferred pages initialised in 65ms [ 0.134112] devtmpfs: initialized [ 0.135213] x86/mm: Memory block size: 128MB [ 0.138735] gcov: version magic: 0x41383552 [ 0.142298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144302] pinctrl core: initialized pinctrl subsystem [ 0.145206] [ 0.145762] ************************************************************* [ 0.146015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147014] ** ** [ 0.148011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149009] ** ** [ 0.150010] ** This means that this kernel is built to expose internal ** [ 0.151009] ** IOMMU data structures, which may compromise security on ** [ 0.152007] ** your system. ** [ 0.153014] ** ** [ 0.154012] ** If you see this message and you are not debugging the ** [ 0.155012] ** kernel, report this immediately to your vendor! ** [ 0.156012] ** ** [ 0.157020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158020] ************************************************************* [ 0.159623] NET: Registered protocol family 16 [ 0.161476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173109] cpuidle: using governor menu [ 0.174940] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177505] PCI: Using configuration type 1 for base access [ 0.179242] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.188102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193222] cryptd: max_cpu_qlen set to 1000 [ 0.196389] ACPI: Added _OSI(Module Device) [ 0.197014] ACPI: Added _OSI(Processor Device) [ 0.199013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206418] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213011] ACPI: Interpreter enabled [ 0.214048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215009] ACPI: Using IOAPIC for interrupt routing [ 0.216584] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219281] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228405] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.239064] acpiphp: Slot [2] registered [ 0.239862] acpiphp: Slot [5] registered [ 0.241112] acpiphp: Slot [6] registered [ 0.241995] acpiphp: Slot [7] registered [ 0.243112] acpiphp: Slot [8] registered [ 0.243991] acpiphp: Slot [9] registered [ 0.245073] acpiphp: Slot [10] registered [ 0.245889] acpiphp: Slot [3] registered [ 0.247051] acpiphp: Slot [4] registered [ 0.248075] acpiphp: Slot [11] registered [ 0.249079] acpiphp: Slot [12] registered [ 0.250063] acpiphp: Slot [13] registered [ 0.252091] acpiphp: Slot [14] registered [ 0.253164] acpiphp: Slot [15] registered [ 0.254108] acpiphp: Slot [16] registered [ 0.256153] acpiphp: Slot [17] registered [ 0.257101] acpiphp: Slot [18] registered [ 0.259124] acpiphp: Slot [19] registered [ 0.260110] acpiphp: Slot [20] registered [ 0.261109] acpiphp: Slot [21] registered [ 0.263140] acpiphp: Slot [22] registered [ 0.264099] acpiphp: Slot [23] registered [ 0.266098] acpiphp: Slot [24] registered [ 0.267373] acpiphp: Slot [25] registered [ 0.269102] acpiphp: Slot [26] registered [ 0.270105] acpiphp: Slot [27] registered [ 0.272107] acpiphp: Slot [28] registered [ 0.273110] acpiphp: Slot [29] registered [ 0.274089] acpiphp: Slot [30] registered [ 0.276124] acpiphp: Slot [31] registered [ 0.277063] PCI host bridge to bus 0000:00 [ 0.279028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.281037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.283020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.286025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.288027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.291027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.293188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.296029] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.299277] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.308019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313008] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.314013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.316015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.317011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.319375] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323264] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328801] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.349024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361506] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.372017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.383053] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.416803] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.422018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.430019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.459573] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.466020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.474019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.494029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.504776] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.523022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.533023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.556322] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.562015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.568016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.581017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591279] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.603023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.620020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.632189] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.633259] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.635271] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.636239] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.637119] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.643044] iommu: Default domain type: Passthrough [ 0.644402] SCSI subsystem initialized [ 0.646079] ACPI: bus type USB registered [ 0.647247] usbcore: registered new interface driver usbfs [ 0.649065] usbcore: registered new interface driver hub [ 0.650096] usbcore: registered new device driver usb [ 0.652140] pps_core: LinuxPPS API ver. 1 registered [ 0.653012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.655049] PTP clock support registered [ 0.657052] EDAC MC: Ver: 3.0.0 [ 0.658528] PCI: Using ACPI for IRQ routing [ 0.659677] NetLabel: Initializing [ 0.661012] NetLabel: domain hash size = 128 [ 0.662009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.663069] NetLabel: unlabeled traffic allowed by default [ 0.666043] vgaarb: loaded [ 0.666630] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.668009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.674000] clocksource: Switched to clocksource kvm-clock [ 0.787040] VFS: Disk quotas dquot_6.6.0 [ 0.789051] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.792167] *** VALIDATE ramfs *** [ 0.793807] *** VALIDATE hugetlbfs *** [ 0.795527] pnp: PnP ACPI init [ 0.797925] pnp: PnP ACPI: found 6 devices [ 0.821845] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.824672] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.826403] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.828814] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.831292] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.832960] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.834931] NET: Registered protocol family 2 [ 0.836645] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840340] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.842988] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.847518] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.850406] TCP: Hash tables configured (established 65536 bind 65536) [ 0.852706] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.854938] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857418] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.859838] NET: Registered protocol family 1 [ 0.862498] RPC: Registered named UNIX socket transport module. [ 0.864649] RPC: Registered udp transport module. [ 0.866306] RPC: Registered tcp transport module. [ 0.867901] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.870231] NET: Registered protocol family 44 [ 0.871850] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.874086] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.876101] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.878458] PCI: CLS 0 bytes, default 64 [ 0.880234] Unpacking initramfs... [ 2.355555] debug: unmapping init [mem 0xffff9b113cc54000-0xffff9b113ffbffff] [ 2.359578] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.361924] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.364772] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.839757] Initialise system trusted keyrings [ 2.841081] Key type blacklist registered [ 2.842735] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.851122] zbud: loaded [ 2.853683] *** VALIDATE nfs *** [ 2.854779] *** VALIDATE nfs4 *** [ 2.856157] pstore: using deflate compression [ 2.858786] Platform Keyring initialized [ 2.939398] NET: Registered protocol family 38 [ 2.941345] Key type asymmetric registered [ 2.942566] Asymmetric key parser 'x509' registered [ 2.943757] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.945597] io scheduler mq-deadline registered [ 2.947424] io scheduler kyber registered [ 2.949414] io scheduler bfq registered [ 2.951620] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.955237] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.958885] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.962510] ACPI: Power Button [PWRF] [ 2.967739] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.973599] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.984730] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.990524] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.002901] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.029604] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.056124] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.060554] Non-volatile memory driver v1.3 [ 3.061823] Linux agpgart interface v0.103 [ 3.089101] virtio_blk virtio1: [vda] 155184 512-byte logical blocks (79.5 MB/75.8 MiB) [ 3.091816] vda: detected capacity change from 0 to 79454208 [ 3.108248] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.111623] vdb: detected capacity change from 0 to 1073741824 [ 3.130730] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.133065] vdc: detected capacity change from 0 to 2621440000 [ 3.148319] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.150643] vdd: detected capacity change from 0 to 2621440000 [ 3.165689] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.168801] vde: detected capacity change from 0 to 4294967296 [ 3.182022] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.184178] vdf: detected capacity change from 0 to 4294967296 [ 3.188915] libphy: Fixed MDIO Bus: probed [ 3.193288] usbcore: registered new interface driver usbserial_generic [ 3.195284] usbserial: USB Serial support registered for generic [ 3.197025] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.200569] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.201917] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.203758] mousedev: PS/2 mouse device common for all mice [ 3.206653] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.207283] rtc_cmos 00:05: RTC can wake from S4 [ 3.213657] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.215878] rtc_cmos 00:05: registered as rtc0 [ 3.219133] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.219661] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.224301] intel_pstate: CPU model not supported [ 3.226644] hid: raw HID events driver (C) Jiri Kosina [ 3.228228] usbcore: registered new interface driver usbhid [ 3.229858] usbhid: USB HID core driver [ 3.230955] drop_monitor: Initializing network drop monitor service [ 3.233049] Initializing XFRM netlink socket [ 3.234524] NET: Registered protocol family 10 [ 3.237190] Segment Routing with IPv6 [ 3.238240] NET: Registered protocol family 17 [ 3.239826] mpls_gso: MPLS GSO support [ 3.245330] RAS: Correctable Errors collector initialized. [ 3.247135] AVX version of gcm_enc/dec engaged. [ 3.248817] AES CTR mode by8 optimization enabled [ 3.315338] sched_clock: Marking stable (3315315410, 0)->(4162826231, -847510821) [ 3.319232] registered taskstats version 1 [ 3.320879] Loading compiled-in X.509 certificates [ 3.322877] zswap: loaded using pool lzo/zbud [ 3.344566] Key type big_key registered [ 3.353156] Key type encrypted registered [ 3.354933] ima: No TPM chip found, activating TPM-bypass! [ 3.357099] ima: Allocated hash algorithm: sha1 [ 3.358777] ima: No architecture policies found [ 3.360460] evm: Initialising EVM extended attributes: [ 3.362882] evm: security.selinux [ 3.364075] evm: security.ima [ 3.365208] evm: security.capability [ 3.366627] evm: HMAC attrs: 0x1 [ 3.368887] rtc_cmos 00:05: setting system clock to 2026-09-09 08:14:23 UTC (1788941663) [ 3.375240] debug: unmapping init [mem 0xffffffff91403000-0xffffffff915fffff] [ 3.378332] debug: unmapping init [mem 0xffffffff90182000-0xffffffff90458fff] [ 3.387091] Write protecting the kernel read-only data: 28672k [ 3.390387] debug: unmapping init [mem 0xffffffff8e803000-0xffffffff8e9fffff] [ 3.392741] debug: unmapping init [mem 0xffffffff8f114000-0xffffffff8f1fffff] [ 3.422819] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.430747] systemd[1]: Detected virtualization kvm. [ 3.432180] systemd[1]: Detected architecture x86-64. [ 3.433537] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.461518] systemd[1]: No hostname configured. [ 3.462735] systemd[1]: Set hostname to . [ 3.464186] random: systemd: uninitialized urandom read (16 bytes read) [ 3.465976] systemd[1]: Initializing machine ID from random generator. [ 3.506316] random: ln: uninitialized urandom read (6 bytes read) [ 3.576872] random: systemd: uninitialized urandom read (16 bytes read) [ 3.579232] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.584197] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.589415] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.230398] device-mapper: uevent: version 1.0.3 [ 4.232391] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.922100] virtio_net virtio0 ens2: renamed from eth0 [ 4.936321] random: fast init done [ 5.014708] scsi host0: ata_piix [ 5.038046] scsi host1: ata_piix [ 5.039685] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.042169] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.402804] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.871558] random: crng init done [ 9.872762] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.261524] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.427711] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.650281] SELinux: Disabled at runtime. [ 11.718245] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.726389] systemd[1]: Detected virtualization kvm. [ 11.728856] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.245301] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.248754] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.254423] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.258422] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.261736] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.271975] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.280786] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ 12.322990] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.721317] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.055310] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.094718] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.190777] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.226870] EDAC sbridge: Ver: 1.1.2 [ 15.142390] Key type dns_resolver registered [ 15.469955] NFS: Registering the id_resolver key type [ 15.471636] Key type id_resolver registered [ 15.473151] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg249-server login: [ 51.554512] libcfs: loading out-of-tree module taints kernel. [ 51.641270] Key type ._llcrypt registered [ 51.642890] Key type .llcrypt registered [ 51.738625] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 51.950024] hrtimer: interrupt took 3932361 ns [ 73.593693] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 75.803482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.836415] alg: No test for adler32 (adler32-zlib) [ 77.625779] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 78.970425] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 80.816268] Key type lgssc registered [ 83.087700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.594315] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.736397] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 160.048619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 160.074920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.274948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.338697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.456823] Lustre: lustre-MDT0000: new disk, initializing [ 161.603793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.627685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.090886] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.721362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.828249] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.864905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.869301] Lustre: Skipped 1 previous similar message [ 184.947562] Lustre: lustre-MDT0001: new disk, initializing [ 185.022192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.105709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.129042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.255886] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.608932] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 208.316857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.694434] Lustre: lustre-OST0000: new disk, initializing [ 208.706637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 208.724867] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.818092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.539387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 210.553296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.643199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.261713] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.509758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.732687] Lustre: lustre-OST0001: new disk, initializing [ 233.739805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 233.745909] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.871910] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.747372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.758566] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.924891] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.261259] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.087731] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.761718] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.602648] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.931505] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing yml_node [ 283.258239] Lustre: DEBUG MARKER: Client: 2.17.57.55 [ 286.385479] Lustre: DEBUG MARKER: MDS: 2.17.57.55 [ 289.743247] Lustre: DEBUG MARKER: OSS: 2.17.57.55 [ 291.791279] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 04:19:10 EDT 2026 [ 310.371842] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 311.858862] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.795496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.798240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 333.806138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.828860] Lustre: Skipped 3 previous similar messages [ 337.271466] Lustre: server umount lustre-MDT0000 complete [ 341.169936] LustreError: 7466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942001 with bad export cookie 13316316257632239776 [ 341.173463] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 344.055162] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.056745] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.062282] Lustre: Skipped 4 previous similar messages [ 344.075094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.105073] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 344.120992] Lustre: Skipped 1 previous similar message [ 346.081938] LustreError: 11085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.090061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.104515] LustreError: 11085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 346.130064] Lustre: Skipped 1 previous similar message [ 347.856720] Lustre: server umount lustre-MDT0001 complete [ 353.672788] Lustre: server umount lustre-OST0000 complete [ 359.712087] Lustre: server umount lustre-OST0001 complete [ 367.279927] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 377.165456] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 412.297969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.747865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.790918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.919310] Lustre: lustre-MDT0000: new disk, initializing [ 413.032733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.052419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.329035] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 430.417901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.545752] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.686663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.699150] Lustre: Skipped 1 previous similar message [ 430.836896] Lustre: lustre-MDT0001: new disk, initializing [ 431.031221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 431.092548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 431.108559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.861897] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 447.131997] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.894687] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.405102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.834495] Lustre: lustre-OST0000: new disk, initializing [ 460.838584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 460.844973] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 460.977657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.730367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.737456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.912888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 467.062560] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 477.519545] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 482.022813] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 482.307363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 486.704014] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.915154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.548677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.551049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.574733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.575354] Lustre: Skipped 1 previous similar message [ 493.599294] Lustre: Skipped 1 previous similar message [ 498.661663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.681014] Lustre: Skipped 1 previous similar message [ 503.263132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 503.655942] Lustre: server umount lustre-OST0000 complete [ 512.992595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 513.002932] LustreError: Skipped 1 previous similar message [ 513.009516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.027901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.255279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 524.814747] Lustre: server umount lustre-MDT0000 complete [ 528.362904] LustreError: 15084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.381825] LustreError: 15084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 528.731906] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942188 with bad export cookie 13316316257632241309 [ 528.738862] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.741184] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 529.108754] Lustre: server umount lustre-MDT0001 complete [ 535.547934] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:23:14 (1788942194) [ 556.510073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.195852] LustreError: 19924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.279509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.378931] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 562.662920] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.133992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.124470] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 585.202825] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 591.825779] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 593.989818] Lustre: Failing over lustre-MDT0000 [ 594.219171] Lustre: server umount lustre-MDT0000 complete [ 597.093753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.106230] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.124477] Lustre: Skipped 2 previous similar messages [ 597.143502] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 613.199097] Lustre: 3641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788942257/real 1788942257] req@ffff9b108772a680 x1875841373084416/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788942273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 613.224852] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.235620] LustreError: 21128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.255814] LustreError: 21128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 614.037546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.472136] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb8cd0f155cedcc23 [ 623.482892] Lustre: MGC192.168.202.149@tcp: Connection restored to 0@lo (at 0@lo) [ 624.024423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.035686] Lustre: Skipped 1 previous similar message [ 624.195466] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 629.225997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 629.230745] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 629.248913] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 629.310094] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 641.092163] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 647.648937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.652801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.659135] Lustre: Skipped 1 previous similar message [ 647.663551] Lustre: Skipped 6 previous similar messages [ 651.483250] Lustre: server umount lustre-MDT0000 complete [ 652.772075] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.798495] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 655.521104] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942315 with bad export cookie 13316316257632242723 [ 655.542753] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.270880] Lustre: server umount lustre-MDT0001 complete [ 666.244330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.705409] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 682.043444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.836894] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 695.190654] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 700.432856] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 702.659592] Lustre: Failing over lustre-MDT0000 [ 702.673347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.684277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.688471] Lustre: Skipped 2 previous similar messages [ 702.990320] Lustre: server umount lustre-MDT0000 complete [ 718.312459] LustreError: 23262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.343218] LustreError: 23262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 722.911416] Lustre: 3641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788942367/real 1788942367] req@ffff9b11a6e65c00 x1875841373113856/t0(0) o400->MGC192.168.202.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788942383 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 722.990166] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.592439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.698805] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 738.793706] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 738.805556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 738.812185] Lustre: Skipped 1 previous similar message [ 738.822141] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 738.852449] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 750.422345] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 754.850676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.862492] Lustre: Skipped 1 previous similar message [ 759.266360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 761.322349] Lustre: server umount lustre-MDT0000 complete [ 765.349483] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942425 with bad export cookie 13316316257632243661 [ 765.354756] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.357396] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 765.797511] Lustre: server umount lustre-MDT0001 complete [ 777.122312] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:27:15 (1788942435) [ 798.484485] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 808.201019] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 849.088925] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 864.523862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.012893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 865.075447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 865.238405] Lustre: lustre-MDT0000: new disk, initializing [ 865.416057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.437452] Lustre: Skipped 3 previous similar messages [ 865.506692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 873.399869] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 883.988498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.231288] Lustre: 29648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 884.795622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 890.649287] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 901.075603] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 907.698524] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 914.729224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 915.131748] Lustre: 31621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 916.997867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 917.066133] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 917.079708] Lustre: Skipped 1 previous similar message [ 917.101314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 917.116323] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 922.419936] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 931.848926] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 938.365203] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 957.925446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.931745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 957.935607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 957.935616] Lustre: Skipped 4 previous similar messages [ 957.951705] Lustre: Skipped 2 previous similar messages [ 957.975037] LustreError: Skipped 1 previous similar message [ 963.411593] Lustre: server umount lustre-OST0000 complete [ 973.473871] Lustre: server umount lustre-MDT0000 complete [ 978.002738] LustreError: 31620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942638 with bad export cookie 13316316257632244067 [ 978.023599] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.032649] LustreError: 31620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 978.990852] Lustre: server umount lustre-MDT0001 complete [ 999.457823] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1003.207729] Key type lgssc unregistered [ 1003.450822] LNet: 33624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.467043] LNetError: 33624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.489553] LNet: Removed LNI 192.168.202.149@tcp [ 1004.269205] Key type .llcrypt unregistered [ 1004.271465] Key type ._llcrypt unregistered [ 1030.491984] Key type ._llcrypt registered [ 1030.498788] Key type .llcrypt registered [ 1030.717149] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1032.600516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1032.646713] alg: No test for adler32 (adler32-zlib) [ 1033.776840] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1034.046729] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1035.719209] Key type lgssc registered [ 1037.758307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1046.658983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1046.680353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.438562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1053.775087] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1058.799066] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.815583] LustreError: 35045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1063.906803] LustreError: 35046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.660868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.268435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1070.612691] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1079.955273] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1086.930738] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1094.790880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1095.281852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.189430] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1110.641909] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1111.346558] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1118.862859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1136.761811] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:33:15 (1788942795) [ 1138.976460] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1141.009406] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1148.302260] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:33:27 (1788942807) [ 1157.230309] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:33:36 (1788942816) [ 1158.342721] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1167.369447] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1167.379378] Lustre: Skipped 2 previous similar messages [ 1169.403034] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:33:47 (1788942827) [ 1171.436993] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1171.448594] Lustre: Skipped 2 previous similar messages [ 1180.546566] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1180.551547] Lustre: Skipped 4 previous similar messages [ 1196.219481] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:34:15 (1788942855) [ 1199.205840] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1203.290963] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1213.734974] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:34:32 (1788942872) [ 1215.724736] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1215.734086] Lustre: Skipped 6 previous similar messages [ 1230.474247] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:34:48 (1788942888) [ 1247.207763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1247.208458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.212340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.734292] Lustre: server umount lustre-OST0000 complete [ 1257.440397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.441466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.445515] Lustre: Skipped 2 previous similar messages [ 1257.452286] Lustre: Skipped 2 previous similar messages [ 1262.084428] Lustre: server umount lustre-MDT0000 complete [ 1262.563975] LustreError: 36638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.588230] LustreError: 36638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1266.047200] LustreError: 40095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942926 with bad export cookie 1371395051079263400 [ 1266.051730] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1266.058529] LustreError: 40095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1266.521459] Lustre: server umount lustre-MDT0001 complete [ 1277.934531] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1280.438759] Key type lgssc unregistered [ 1280.761520] LNet: 42925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.768537] LNetError: 42925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.786857] LNet: Removed LNI 192.168.202.149@tcp [ 1281.689629] Key type .llcrypt unregistered [ 1281.697168] Key type ._llcrypt unregistered [ 1301.077260] Key type ._llcrypt registered [ 1301.080843] Key type .llcrypt registered [ 1301.205574] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 1318.914416] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1319.918359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1319.941326] alg: No test for adler32 (adler32-zlib) [ 1321.120930] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1321.359791] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1323.039170] Key type lgssc registered [ 1324.590672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1361.535248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1361.560267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.982236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1363.021769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1363.213635] Lustre: lustre-MDT0000: new disk, initializing [ 1363.375847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.389911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1367.741617] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1379.673766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.883104] Lustre: 46083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1379.992301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1379.996743] Lustre: Skipped 1 previous similar message [ 1380.119954] Lustre: lustre-MDT0001: new disk, initializing [ 1380.254405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.282947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1380.293450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1384.787378] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1396.582341] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1402.318530] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1410.775407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.341769] Lustre: lustre-OST0000: new disk, initializing [ 1411.349302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1411.359747] Lustre: 48049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1411.505824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.902935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1419.911946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1420.042775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1420.276133] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1431.959401] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1437.324342] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1437.711744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1443.099182] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1443.275772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1445.856959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.870630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.902675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.905330] Lustre: Skipped 1 previous similar message [ 1450.995821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.006842] Lustre: Skipped 1 previous similar message [ 1451.696226] Lustre: server umount lustre-OST0000 complete [ 1456.099288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.125220] Lustre: Skipped 1 previous similar message [ 1456.135078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.149408] Lustre: Skipped 1 previous similar message [ 1461.216637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.229363] Lustre: Skipped 1 previous similar message [ 1462.498100] Lustre: server umount lustre-MDT0000 complete [ 1467.949289] LustreError: 46877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943128 with bad export cookie 15341827196450247824 [ 1467.957769] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.975283] LustreError: 46877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1468.898891] Lustre: server umount lustre-MDT0001 complete [ 1481.101448] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:38:59 (1788943139) [ 1490.679347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.341879] LustreError: 50512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.367513] LustreError: 50512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1491.471724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.548030] LustreError: 50513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.465717] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1501.667882] LustreError: 50512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.797425] LustreError: 50513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.689915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.126530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1515.198213] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1524.322920] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1531.217445] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1538.183684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1538.877262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.592661] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1556.052449] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1565.684707] LustreError: 53043:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1566.737359] Lustre: Modifying parameter general.timeout=1 in log params [ 1568.524901] Lustre: Modifying parameter general.timeout=3 in log params [ 1568.528511] Lustre: Skipped 1 previous similar message [ 1570.604213] Lustre: Modifying parameter general.timeout=5 in log params [ 1570.609211] Lustre: Skipped 1 previous similar message [ 1575.443718] Lustre: Modifying parameter general.timeout=10 in log params [ 1575.446965] Lustre: Skipped 4 previous similar messages [ 1584.126215] Lustre: Modifying parameter general.timeout=19 in log params [ 1584.133789] Lustre: Skipped 8 previous similar messages [ 1600.598394] Lustre: Modifying parameter general.timeout=35 in log params [ 1600.602399] Lustre: Skipped 15 previous similar messages [ 1632.930671] Lustre: Modifying parameter general.timeout=68 in log params [ 1632.937725] Lustre: Skipped 32 previous similar messages [ 1702.073993] Lustre: Disabling parameter general.timeout= in log params [ 1702.083704] Lustre: Skipped 61 previous similar messages [ 1705.620703] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:42:44 (1788943364) [ 1708.542553] Lustre: 59993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1710.309797] Lustre: 60092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1712.359096] Lustre: 60191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1718.169447] Lustre: 60505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1718.175639] Lustre: 60505:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1745.576954] Lustre: 61867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1745.607527] LustreError: 61867:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1750.953546] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:43:28 (1788943408) [ 1765.347865] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.352651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1765.365608] Lustre: Skipped 1 previous similar message [ 1765.372915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.381991] LustreError: Skipped 1 previous similar message [ 1765.392819] Lustre: Skipped 2 previous similar messages [ 1770.377224] Lustre: server umount lustre-MDT0000 complete [ 1770.470897] LustreError: 51241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1774.165709] LustreError: 50492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943434 with bad export cookie 15341827196450248706 [ 1774.166325] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1774.209565] LustreError: 50492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1774.408914] Lustre: server umount lustre-MDT0001 complete [ 1788.952510] Lustre: server umount lustre-OST0000 complete [ 1817.329896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.640831] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1817.990258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.515665] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1833.583809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.695676] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1833.741252] Lustre: 64399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1839.881192] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1847.973506] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1854.037383] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1861.717910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.037715] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1862.449289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.456793] Lustre: Skipped 1 previous similar message [ 1870.134301] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1880.241335] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1881.877987] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1881.886599] Lustre: Skipped 23 previous similar messages [ 1881.937526] Lustre: 66965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1890.276062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.276331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.305422] Lustre: Skipped 2 previous similar messages [ 1890.310814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.315854] Lustre: Skipped 1 previous similar message [ 1896.253319] Lustre: server umount lustre-OST0000 complete [ 1900.515565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.516195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1900.533377] Lustre: Skipped 2 previous similar messages [ 1906.863925] Lustre: server umount lustre-MDT0000 complete [ 1910.752531] LustreError: 64404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.781466] LustreError: 64404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1910.968047] LustreError: 64390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943571 with bad export cookie 15341827196450251800 [ 1910.986269] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1910.990469] LustreError: 64390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1911.481595] Lustre: server umount lustre-MDT0001 complete [ 1922.756431] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 1925.476833] Key type lgssc unregistered [ 1925.765593] LNet: 68180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.772955] LNetError: 68180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.830878] LNet: Removed LNI 192.168.202.149@tcp [ 1927.808128] Key type .llcrypt unregistered [ 1927.810236] Key type ._llcrypt unregistered [ 1953.114208] Key type ._llcrypt registered [ 1953.117782] Key type .llcrypt registered [ 1953.380277] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 1955.382139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.594750] alg: No test for adler32 (adler32-zlib) [ 1956.772975] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 1957.206612] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 1958.927201] Key type lgssc registered [ 1960.189842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1970.374723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1970.405675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.974668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.973763] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1982.448715] LustreError: 69597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.469348] LustreError: 69597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1986.317742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.844483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.577664] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 1999.584529] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2005.520516] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2012.073246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2012.600110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.904759] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2028.992888] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2037.334533] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2047.969049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2047.976548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.988176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2051.552529] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.557779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2051.565845] Lustre: Skipped 1 previous similar message [ 2051.574433] Lustre: Skipped 1 previous similar message [ 2052.979437] Lustre: server umount lustre-MDT0000 complete [ 2056.648711] LustreError: 71449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943716 with bad export cookie 4362570941540983513 [ 2056.662673] LustreError: 71449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2056.668228] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.697254] LustreError: 69591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2056.706450] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.746169] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2056.757633] Lustre: Skipped 1 previous similar message [ 2057.055845] Lustre: server umount lustre-MDT0001 complete [ 2063.175921] Lustre: server umount lustre-OST0000 complete [ 2093.042028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.248758] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2093.273857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2093.575466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.430437] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2107.664310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.769747] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2107.795620] Lustre: 74469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2108.509890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2113.735161] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2121.721175] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.389806] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2135.840388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2136.156463] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2136.615790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.109589] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2152.389179] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2154.030238] Lustre: 77042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2156.023642] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2162.144373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2162.150657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.161503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2167.266897] Lustre: server umount lustre-OST0000 complete [ 2172.899449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.904916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.917710] Lustre: Skipped 2 previous similar messages [ 2172.943302] Lustre: Skipped 2 previous similar messages [ 2182.117484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.131949] Lustre: Skipped 3 previous similar messages [ 2185.183478] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2185.402484] Lustre: server umount lustre-MDT0000 complete [ 2187.232522] LustreError: 74479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.254371] LustreError: 74479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2189.234017] LustreError: 74461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943849 with bad export cookie 4362570941540984360 [ 2189.235367] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.246893] LustreError: 74461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2189.764568] Lustre: server umount lustre-MDT0001 complete [ 2201.370086] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2205.011935] Key type lgssc unregistered [ 2205.327592] LNet: 78260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.341261] LNetError: 78260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.371790] LNet: Removed LNI 192.168.202.149@tcp [ 2206.368838] Key type .llcrypt unregistered [ 2206.370553] Key type ._llcrypt unregistered [ 2230.814272] Key type ._llcrypt registered [ 2230.817668] Key type .llcrypt registered [ 2230.918404] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2231.947584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.005333] alg: No test for adler32 (adler32-zlib) [ 2233.068610] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2233.254256] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2234.911158] Key type lgssc registered [ 2236.134953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2244.205785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2244.230048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.794943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.467868] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2255.329349] LustreError: 79673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.346465] LustreError: 79673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2259.736968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.418179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2264.333169] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2270.431318] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2275.117888] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2281.807246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2282.279676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.034481] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2298.390962] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2306.314772] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2307.711909] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2309.015381] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2311.329506] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2311.340886] Lustre: Skipped 1 previous similar message [ 2316.443609] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2316.451719] Lustre: Skipped 4 previous similar messages [ 2325.129229] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2325.136392] Lustre: Skipped 9 previous similar messages [ 2341.747171] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2341.752898] Lustre: Skipped 17 previous similar messages [ 2374.512796] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2374.518221] Lustre: Skipped 38 previous similar messages [ 2404.537470] Lustre: 87769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2407.211330] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:54:26 (1788944066) [ 2409.017265] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2410.956731] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:54:29 (1788944069) [ 2418.656255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2418.660320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.667083] LustreError: Skipped 1 previous similar message [ 2418.671122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2418.703302] Lustre: Skipped 1 previous similar message [ 2421.669156] Lustre: server umount lustre-OST0000 complete [ 2428.900189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.935515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2428.948271] Lustre: Skipped 2 previous similar messages [ 2431.892980] Lustre: server umount lustre-MDT0000 complete [ 2434.017202] LustreError: 81676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.047927] LustreError: 81676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2436.462190] LustreError: 79655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944096 with bad export cookie 15253313870707627758 [ 2436.464405] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.472255] LustreError: 79655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2436.964974] Lustre: server umount lustre-MDT0001 complete [ 2451.446963] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2455.505615] Key type lgssc unregistered [ 2455.808086] LNet: 89021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2455.823489] LNetError: 89021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2455.849406] LNet: Removed LNI 192.168.202.149@tcp [ 2456.717920] Key type .llcrypt unregistered [ 2456.719701] Key type ._llcrypt unregistered [ 2461.870613] Key type ._llcrypt registered [ 2461.873427] Key type .llcrypt registered [ 2461.995557] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2466.792980] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules [ 2468.376936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.431487] alg: No test for adler32 (adler32-zlib) [ 2469.507987] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2469.825061] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2471.607440] Key type lgssc registered [ 2473.233556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2474.587730] LustreError: 90240:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2490.514905] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2495.965536] LustreError: 90240:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2500.389828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2500.421051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.879669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.348284] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2511.337869] LustreError: 91247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.357410] LustreError: 91247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2514.669817] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:56:13 (1788944173) [ 2516.454672] LustreError: 91248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.654420] LustreError: 91279:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2518.666955] LustreError: 91279:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2518.676076] LustreError: 91279:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2519.066325] Lustre: server umount lustre-MDT0000 complete [ 2531.328715] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 2534.334130] Key type lgssc unregistered [ 2534.716293] LNet: 92774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.729104] LNetError: 92774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.783503] LNet: Removed LNI 192.168.202.149@tcp [ 2536.806623] Key type .llcrypt unregistered [ 2536.808886] Key type ._llcrypt unregistered [ 2557.359654] Key type ._llcrypt registered [ 2557.362699] Key type .llcrypt registered [ 2557.509679] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2558.104784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.122441] alg: No test for adler32 (adler32-zlib) [ 2559.123968] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 2559.292730] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 2560.935188] Key type lgssc registered [ 2561.877841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2569.257361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2569.287717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.818726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.772917] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2579.940391] LustreError: 93994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.952078] LustreError: 93994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2582.472571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.088784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2587.678320] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2594.550376] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2600.521429] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2607.465774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2607.599627] LustreError: 95824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.261809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.820435] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2622.849254] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2669.179396] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:58:47 (1788944327) [ 2671.052652] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2673.028527] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:58:51 (1788944331) [ 2677.223108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.244099] Lustre: Skipped 2 previous similar messages [ 2677.258368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.264393] Lustre: Skipped 2 previous similar messages [ 2682.338884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.350172] Lustre: Skipped 2 previous similar messages [ 2682.739743] Lustre: server umount lustre-MDT0000 complete [ 2686.410824] LustreError: 94735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944346 with bad export cookie 17031390830265993162 [ 2686.415599] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.425579] LustreError: 94735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2686.685700] Lustre: server umount lustre-MDT0001 complete [ 2701.146152] Lustre: server umount lustre-OST0000 complete [ 2714.126026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.703636] LustreError: 98276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.065670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.230790] LustreError: 98277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.256881] LustreError: 98277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2720.435892] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2724.335330] LustreError: 98276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.282378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.414362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2736.248846] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2743.740508] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2749.010485] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2760.751316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2761.035799] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2761.045409] LustreError: 98265:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2761.091132] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2761.109892] LustreError: 100359:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2761.123210] LustreError: 100359:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2761.128731] LustreError: 100359:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2761.230507] Lustre: server umount lustre-OST0000 complete [ 2761.233547] LustreError: 100359:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2769.123161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2769.397025] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2769.409804] LustreError: 98266:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2769.425845] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2769.443114] LustreError: 100820:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2769.454115] LustreError: 100820:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2769.459405] LustreError: 100820:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2769.565362] Lustre: server umount lustre-OST0000 complete [ 2769.567839] LustreError: 100820:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2778.553207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.763822] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2778.768592] Lustre: Found index 0 for lustre-OST0000, updating log [ 2778.787432] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2778.947442] Lustre: lustre-OST0000: new disk, initializing [ 2778.960356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2778.973212] Lustre: 101356:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2779.267683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.655066] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2788.761365] Lustre: Failing over lustre-OST0000 [ 2788.940606] Lustre: server umount lustre-OST0000 complete [ 2794.985272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2795.000339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2795.014429] Lustre: Skipped 1 previous similar message [ 2800.098697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.107463] Lustre: Skipped 1 previous similar message [ 2803.280141] Lustre: server umount lustre-MDT0000 complete [ 2806.242071] LustreError: 98277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.253174] LustreError: 98277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2808.119076] LustreError: 98256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944468 with bad export cookie 17031390830265995906 [ 2808.123718] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2808.129194] LustreError: 98256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2808.521267] Lustre: server umount lustre-MDT0001 complete [ 2816.600529] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:01:15 (1788944475) [ 2832.190734] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2842.012940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.383186] LustreError: 103715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.446439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2846.774615] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2854.370648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.707799] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2865.713152] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2872.562724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.387857] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2879.536159] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2879.541384] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2879.553755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2879.573667] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2884.584897] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2884.609870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2887.156819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.244618] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2887.307122] Lustre: lustre-OST0001: new disk, initializing [ 2887.309903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2887.315064] Lustre: 106033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2887.548297] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 2887.576481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2887.591569] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2887.600782] Lustre: Skipped 2 previous similar messages [ 2888.641335] Lustre: lustre-OST0000: Recovery over after 0:09, of 1 clients 1 recovered and 0 were evicted. [ 2888.643154] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2888.657939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2893.926155] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2897.015668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2897.027782] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2897.102973] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2901.530750] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2912.126595] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2916.838564] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2916.849779] LustreError: Skipped 1 previous similar message [ 2916.861386] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.888746] Lustre: Skipped 1 previous similar message [ 2916.895153] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2916.897805] Lustre: Skipped 2 previous similar messages [ 2922.518535] Lustre: server umount lustre-MDT0001 complete [ 2922.997731] LustreError: 103715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.018326] LustreError: 103715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2931.961443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.041358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2932.044632] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2932.083345] Lustre: 103703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2933.624374] LustreError: 107578:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2933.632726] LustreError: 107578:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2933.642487] LustreError: 103707:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2933.744900] LustreError: 107587:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.149@tcp: cfg command failed: rc = -17 [ 2933.763214] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2933.763214] [ 2937.767607] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 2937.859023] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2948.216775] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:03:27 (1788944607) [ 2966.587437] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 2983.029119] Lustre: 109648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3006.082755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3010.035784] Lustre: 110784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3090.911878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3090.923700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.941733] Lustre: Skipped 2 previous similar messages [ 3090.952850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3090.956124] Lustre: Skipped 5 previous similar messages [ 3095.605048] Lustre: server umount lustre-MDT0000 complete [ 3096.549586] LustreError: 103711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.577532] LustreError: 103711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3099.337448] LustreError: 104456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944759 with bad export cookie 17031390830265996837 [ 3099.337659] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3099.345520] LustreError: 104456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3099.552915] Lustre: server umount lustre-MDT0001 complete [ 3113.821530] Lustre: server umount lustre-OST0000 complete [ 3117.023137] Lustre: 93304:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788944761/real 1788944761] req@ffff9b10932d0a80 x1875843973735040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788944777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3117.080468] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.103394] Lustre: Skipped 3 previous similar messages [ 3117.879400] Lustre: server umount lustre-OST0001 complete [ 3123.475973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3133.330152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3143.955497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3154.914160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3176.533500] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3186.833880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.853298] Lustre: lustre-MDT0000: reset Object Index mappings [ 3187.414930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.428447] Lustre: Skipped 1 previous similar message [ 3191.669554] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3200.006134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.019996] Lustre: lustre-MDT0001: reset Object Index mappings [ 3204.920397] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3207.419472] Lustre: 115887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3212.792938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3218.920946] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3225.645637] LustreError: 116241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.672126] LustreError: 116241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3225.691592] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3225.795507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3231.738780] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3232.499613] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3232.750210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3232.761908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3239.667315] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3243.060507] Lustre: 117763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3289.465913] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:09:07 (1788944947) [ 3314.674750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.693327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3314.695923] Lustre: Skipped 4 previous similar messages [ 3320.978519] LustreError: 118121:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff9b108e119900) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3323.096987] Lustre: server umount lustre-MDT0000 complete [ 3327.461185] LustreError: 114721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944987 with bad export cookie 17031390830266043933 [ 3327.466207] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.470210] LustreError: 114721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3333.951662] Lustre: server umount lustre-MDT0001 complete [ 3343.933753] Lustre: server umount lustre-OST0000 complete [ 3353.460946] Lustre: server umount lustre-OST0001 complete [ 3360.552423] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 3370.519220] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3406.149742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3406.359737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3406.381108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3406.443428] Lustre: lustre-MDT0000: new disk, initializing [ 3406.491596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.494872] Lustre: Skipped 3 previous similar messages [ 3406.501217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3410.243671] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3417.142281] Lustre: server umount lustre-MDT0000 complete [ 3427.201879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.461406] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3427.473407] Lustre: 122035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3427.482126] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3432.911635] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3437.377710] Lustre: server umount lustre-MDT0000 complete [ 3446.157648] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:11:44 (1788945104) [ 3448.248585] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3450.208994] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:11:48 (1788945108) [ 3467.253343] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 3477.481879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.064387] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3492.048315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.182737] Lustre: 124082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3492.247619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3492.251109] Lustre: Skipped 1 previous similar message [ 3492.314979] Lustre: lustre-MDT0001: new disk, initializing [ 3492.444040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3492.457167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3497.424227] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3502.426579] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3509.202387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.494729] Lustre: lustre-OST0000: new disk, initializing [ 3509.507205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3509.521510] Lustre: 125662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3510.633059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3510.646951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3510.844236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3516.391918] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3527.492390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3527.662840] Lustre: lustre-OST0001: new disk, initializing [ 3527.666630] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3527.671786] Lustre: 126534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3528.940496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3528.945976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3529.074424] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3534.104086] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 3542.468694] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3545.997475] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3906.604605] Lustre: 132943:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3906.608513] Lustre: 132943:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3906.612267] Lustre: 132943:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3906.615278] Lustre: 132943:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3906.618045] Lustre: 132943:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3906.626695] Lustre: 132943:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4169.521156] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:23:47 (1788945827) [ 4184.548850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.551063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4184.561681] Lustre: Skipped 7 previous similar messages [ 4184.572486] Lustre: Skipped 10 previous similar messages [ 4189.153569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4189.156691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.162433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4190.855476] Lustre: server umount lustre-MDT0000 complete [ 4194.244962] LustreError: 132988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945854 with bad export cookie 17031390830266110286 [ 4194.258306] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.266752] LustreError: 132988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4194.786140] LustreError: 127340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.786916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.805916] LustreError: 127340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4194.825928] Lustre: Skipped 1 previous similar message [ 4199.912401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4199.919040] Lustre: Skipped 5 previous similar messages [ 4200.640515] Lustre: server umount lustre-MDT0001 complete [ 4210.562113] Lustre: server umount lustre-OST0000 complete [ 4220.244956] Lustre: server umount lustre-OST0001 complete [ 4226.472646] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 4236.628091] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 4273.065145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.417280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4273.474837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4273.574385] Lustre: lustre-MDT0000: new disk, initializing [ 4273.664450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.671503] Lustre: Skipped 5 previous similar messages [ 4273.683875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4278.950994] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4289.835067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.971909] Lustre: 135990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.021327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4290.026555] Lustre: Skipped 1 previous similar message [ 4290.103312] Lustre: lustre-MDT0001: new disk, initializing [ 4290.252340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4290.265818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4295.431855] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4304.181211] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4310.069098] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4317.214847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4317.648376] Lustre: lustre-OST0000: new disk, initializing [ 4317.653587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4317.675648] Lustre: 137956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4317.777633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.787936] Lustre: Skipped 1 previous similar message [ 4318.750712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4318.769304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4318.970093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4324.121269] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 4333.462123] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4343.071376] Lustre: lustre-MDD0000: changelog on [ 4345.771263] Lustre: lustre-MDD0001: changelog on [ 6069.117550] Lustre: lustre-MDD0001: changelog off [ 6071.321135] Lustre: 135995:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6072.365575] Lustre: lustre-MDD0000: changelog off [ 6080.484829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6080.498363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.499246] LustreError: Skipped 1 previous similar message [ 6080.503721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6080.538817] Lustre: Skipped 1 previous similar message [ 6083.562083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6083.572081] Lustre: Skipped 2 previous similar messages [ 6088.673993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.684954] Lustre: Skipped 1 previous similar message [ 6093.279174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6093.500073] Lustre: server umount lustre-OST0000 complete [ 6097.888988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6097.894607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.912790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6103.564488] Lustre: server umount lustre-MDT0000 complete [ 6104.035199] LustreError: 136001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.049365] LustreError: 136001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6106.980160] LustreError: 135982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947767 with bad export cookie 17031390830266925527 [ 6106.980980] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6106.986829] LustreError: 135982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6107.310759] Lustre: server umount lustre-MDT0001 complete [ 6117.394635] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 6120.007658] Key type lgssc unregistered [ 6120.377300] LNet: 147074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6120.393352] LNetError: 147074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6121.454742] LNet: Removed LNI 192.168.202.149@tcp [ 6122.579485] Key type .llcrypt unregistered [ 6122.581638] Key type ._llcrypt unregistered [ 6143.125701] Key type ._llcrypt registered [ 6143.127984] Key type .llcrypt registered [ 6143.240551] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 6159.463557] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 6160.851577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6160.904359] alg: No test for adler32 (adler32-zlib) [ 6161.993765] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 6162.205470] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 6163.863284] Key type lgssc registered [ 6164.940439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6195.743449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6195.768818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.071608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6197.116244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6197.225505] Lustre: lustre-MDT0000: new disk, initializing [ 6197.312963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.330228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6201.708353] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6212.354141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.490673] Lustre: 150280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6212.549451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6212.554399] Lustre: Skipped 1 previous similar message [ 6212.648785] Lustre: lustre-MDT0001: new disk, initializing [ 6212.812457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6212.881765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6212.898403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6216.529990] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6225.636414] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6230.141458] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6236.462367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6236.795155] Lustre: lustre-OST0000: new disk, initializing [ 6236.799917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6236.804774] Lustre: 152250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6236.850145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6242.873371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6242.882343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6243.041856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6243.877671] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6253.707270] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6259.794462] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_module obdecho/obdecho [ 6261.555784] LustreError: 153253:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6261.563618] LustreError: 153253:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6263.149530] LustreError: 153349:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6274.532843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6274.546639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.563788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6279.140449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6279.144726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.502896] Lustre: server umount lustre-MDT0000 complete [ 6283.736469] LustreError: 150270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947943 with bad export cookie 2381115750281283645 [ 6283.746903] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6283.749029] LustreError: 150270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6284.049383] Lustre: server umount lustre-MDT0001 complete [ 6298.520236] Lustre: server umount lustre-OST0000 complete [ 6320.351247] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 6330.283510] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 6366.776367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.038610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6367.066574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6367.140537] Lustre: lustre-MDT0000: new disk, initializing [ 6367.291917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.324815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6371.754949] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6384.342417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.558294] Lustre: 157068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6384.617688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6384.621683] Lustre: Skipped 1 previous similar message [ 6384.711395] Lustre: lustre-MDT0001: new disk, initializing [ 6384.844813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6384.917848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6384.934847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6391.083399] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6402.649490] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6408.865493] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6415.534223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6415.867781] Lustre: lustre-OST0000: new disk, initializing [ 6415.884496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6415.898125] Lustre: 159046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6415.989988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.054113] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6424.635231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6424.656720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6424.779729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6431.818198] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6437.033387] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6437.345049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6442.756098] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6443.080135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6445.539625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.550039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6445.553627] Lustre: Skipped 1 previous similar message [ 6445.563056] LustreError: Skipped 1 previous similar message [ 6445.565344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6445.574178] Lustre: Skipped 3 previous similar messages [ 6450.656594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6450.670779] Lustre: Skipped 1 previous similar message [ 6451.405854] Lustre: server umount lustre-OST0000 complete [ 6455.783196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.802203] Lustre: Skipped 2 previous similar messages [ 6455.802495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6461.890244] Lustre: server umount lustre-MDT0000 complete [ 6465.560894] LustreError: 157058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948125 with bad export cookie 2381115750281284604 [ 6465.569895] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6465.572295] LustreError: 157058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6465.997025] Lustre: server umount lustre-MDT0001 complete [ 6469.779927] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:02:08 (1788948128) [ 6471.124686] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6472.889867] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:02:11 (1788948131) [ 6480.201399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6480.682471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6485.083938] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6485.985708] LustreError: 161460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.012636] LustreError: 161460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6490.085104] LustreError: 161461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.880277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.898370] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6505.143781] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6511.291368] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6519.218807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6519.625409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6519.628226] Lustre: Skipped 1 previous similar message [ 6525.530831] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6533.026647] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6543.333856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.344447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6543.361734] Lustre: Skipped 2 previous similar messages [ 6543.385254] Lustre: Skipped 5 previous similar messages [ 6545.304574] Lustre: server umount lustre-MDT0000 complete [ 6548.454032] LustreError: 162188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.473530] LustreError: 162188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6550.501893] LustreError: 161440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948210 with bad export cookie 2381115750281285486 [ 6550.524295] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6553.568996] LustreError: 163953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6553.572557] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.589465] LustreError: 163953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6557.276867] Lustre: server umount lustre-MDT0001 complete [ 6565.603767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6580.372705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6580.651738] LustreError: 165305:0:(llog.c:1655:llog_backup()) MGC192.168.202.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6580.657451] Lustre: 165305:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6583.989376] LustreError: 165310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.002400] LustreError: 165310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6594.167588] LustreError: 165310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.196722] LustreError: 165310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6610.556591] LustreError: 165312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6610.576674] LustreError: 165312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6641.631343] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6641.637637] Lustre: 165305:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6645.218612] LustreError: 165310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.230704] LustreError: 165310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6682.618322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6687.730840] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6688.029284] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6697.119942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6709.881719] LustreError: 165310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.902473] LustreError: 165310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6710.246340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6720.498975] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x210b6c28c09aa8c7 [ 6720.521924] Lustre: MGC192.168.202.149@tcp: Connection restored to 0@lo (at 0@lo) [ 6725.816883] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6726.130955] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6733.574210] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6735.057278] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6741.082241] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6742.903971] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6761.952886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6761.955976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.968497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6761.982449] Lustre: Skipped 1 previous similar message [ 6762.004714] Lustre: Skipped 6 previous similar messages [ 6765.950021] Lustre: server umount lustre-MDT0000 complete [ 6769.678487] LustreError: 161442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948429 with bad export cookie 2381115750281291975 [ 6769.695143] LustreError: 161442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6769.704448] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6769.998636] Lustre: server umount lustre-MDT0001 complete [ 6783.912133] Lustre: server umount lustre-OST0000 complete [ 6804.867956] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 6814.653795] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 6849.790394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.950379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6849.975750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6850.029281] Lustre: lustre-MDT0000: new disk, initializing [ 6850.095078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6850.106049] Lustre: Skipped 1 previous similar message [ 6850.123563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6855.374401] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6866.221210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.368166] Lustre: 170417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6866.648640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6871.482992] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6881.919591] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6887.808354] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6895.301293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6895.722275] Lustre: lustre-OST0000: new disk, initializing [ 6895.723963] Lustre: Skipped 1 previous similar message [ 6895.734267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6895.740193] Lustre: Skipped 2 previous similar messages [ 6895.745288] Lustre: 172387:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6902.990460] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6904.915218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6904.928483] Lustre: Skipped 1 previous similar message [ 6904.937245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6905.028694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6913.266592] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6918.298270] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6918.418157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6921.640671] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6921.867571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6925.797759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6925.801691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6925.808612] LustreError: Skipped 1 previous similar message [ 6925.815668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6925.844444] Lustre: Skipped 2 previous similar messages [ 6925.867291] Lustre: Skipped 1 previous similar message [ 6929.707568] Lustre: server umount lustre-OST0000 complete [ 6940.634740] Lustre: server umount lustre-MDT0000 complete [ 6941.158200] LustreError: 170429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6941.179836] LustreError: 170429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6944.646576] LustreError: 170408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948604 with bad export cookie 2381115750281308782 [ 6944.652855] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6944.656693] LustreError: 170408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6944.877092] Lustre: server umount lustre-MDT0001 complete [ 6955.825857] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:10:13 (1788948613) [ 6963.437192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6968.321099] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6976.109107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.525050] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 6991.126161] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6998.960504] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7009.269774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.228771] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7027.888836] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7042.796457] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:11:41 (1788948701) [ 7046.625127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.636376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.658913] Lustre: Skipped 2 previous similar messages [ 7046.665442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.673600] Lustre: Skipped 2 previous similar messages [ 7052.665403] Lustre: server umount lustre-OST0000 complete [ 7063.038070] Lustre: server umount lustre-MDT0000 complete [ 7067.029387] LustreError: 174778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948727 with bad export cookie 2381115750281309664 [ 7067.048491] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.049099] LustreError: 174778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7067.412105] Lustre: server umount lustre-MDT0001 complete [ 7078.506939] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 7081.146435] Key type lgssc unregistered [ 7081.394804] LNet: 178530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7081.400763] LNetError: 178530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7081.411744] LNet: Removed LNI 192.168.202.149@tcp [ 7082.123930] Key type .llcrypt unregistered [ 7082.125875] Key type ._llcrypt unregistered [ 7099.743576] Key type ._llcrypt registered [ 7099.756166] Key type .llcrypt registered [ 7099.872787] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7100.792837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7100.815357] alg: No test for adler32 (adler32-zlib) [ 7102.024644] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 7102.316931] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 7104.007248] Key type lgssc registered [ 7105.098544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7112.692538] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7112.708764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.225164] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7133.343391] LustreError: 179773:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.149@tcp: failed processing log, type 4: rc = -110 [ 7138.464317] LustreError: 179773:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7138.480968] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7138.488252] LustreError: MGC192.168.202.149@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7138.505356] LustreError: 179773:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7138.525223] LustreError: 179773:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7138.539247] LustreError: 179773:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7138.630969] Lustre: server umount lustre-OST0000 complete [ 7138.633273] LustreError: 179773:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7147.513657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.162434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7153.481509] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7159.780424] LustreError: 180275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7159.800811] LustreError: 180275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7162.102712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.440858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7167.325726] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7175.315118] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7181.071640] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7187.870330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.177814] LustreError: 182092:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7188.189227] Lustre: 182092:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.149@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7188.340716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7194.060214] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7201.641490] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7204.859995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7204.867316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7204.900799] Lustre: Skipped 2 previous similar messages [ 7204.926392] Lustre: Skipped 2 previous similar messages [ 7207.971345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7210.602960] Lustre: server umount lustre-MDT0000 complete [ 7213.029116] LustreError: 181003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.063769] LustreError: 181003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7215.223457] LustreError: 180256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948875 with bad export cookie 10355096556409848490 [ 7215.230214] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7215.232621] LustreError: 180256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7216.224679] LustreError: 181005:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001cd51529 x1875848737091968/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7216.703472] Lustre: server umount lustre-MDT0001 complete [ 7232.194486] Lustre: server umount lustre-OST0000 complete [ 7242.392951] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7254.231898] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7297.097186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.440105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7297.477345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7297.561386] Lustre: lustre-MDT0000: new disk, initializing [ 7297.645587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.674697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7303.024607] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7314.405397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.549750] Lustre: 185624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7314.600714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7314.604570] Lustre: Skipped 1 previous similar message [ 7314.705985] Lustre: lustre-MDT0001: new disk, initializing [ 7314.810445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7314.847603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7314.860729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7319.825660] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7329.901753] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7336.314323] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7343.547382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7343.768167] Lustre: lustre-OST0000: new disk, initializing [ 7343.774617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7343.783821] Lustre: 187591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7343.872391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7350.280900] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7351.414073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7351.422728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7351.579602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7359.467755] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7364.466715] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7364.834612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7369.826546] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7370.056239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7372.257155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.268523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7372.268557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7372.295834] Lustre: Skipped 3 previous similar messages [ 7377.393931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7377.409729] Lustre: Skipped 1 previous similar message [ 7378.383760] Lustre: server umount lustre-OST0000 complete [ 7382.504455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7382.520874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7382.534232] Lustre: Skipped 2 previous similar messages [ 7382.559062] Lustre: Skipped 1 previous similar message [ 7392.737142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.742976] Lustre: Skipped 3 previous similar messages [ 7396.319292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7396.521938] Lustre: server umount lustre-MDT0000 complete [ 7397.857743] LustreError: 186796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7397.866662] LustreError: 186796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7399.933466] LustreError: 185616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949060 with bad export cookie 10355096556409849211 [ 7399.936864] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7399.940935] LustreError: 185616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7400.340493] Lustre: server umount lustre-MDT0001 complete [ 7409.911717] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:17:48 (1788949068) [ 7430.051239] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 7433.012669] Key type lgssc unregistered [ 7433.341255] LNet: 190543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7433.346182] LNetError: 190543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7433.376955] LNet: Removed LNI 192.168.202.149@tcp [ 7434.211102] Key type .llcrypt unregistered [ 7434.212890] Key type ._llcrypt unregistered [ 7453.721417] Key type ._llcrypt registered [ 7453.723673] Key type .llcrypt registered [ 7453.827867] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7455.337633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7455.439407] alg: No test for adler32 (adler32-zlib) [ 7456.664930] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 7457.093245] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 7458.783365] Key type lgssc registered [ 7460.360022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7468.440081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7468.457858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.940553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7474.410875] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7480.291767] LustreError: 191760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7480.306153] LustreError: 191760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7482.470828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.770510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7487.286657] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7496.458516] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7503.819317] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7512.369352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.640266] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7512.652473] LustreError: 193627:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7512.666276] Lustre: 193627:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.149@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7512.903327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.261347] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7525.169468] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7528.940213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7528.943797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.962794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7532.537395] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.552946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7532.553466] Lustre: Skipped 1 previous similar message [ 7532.569494] Lustre: Skipped 2 previous similar messages [ 7537.634091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7537.653047] Lustre: Skipped 2 previous similar messages [ 7542.240204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7542.515359] Lustre: server umount lustre-MDT0000 complete [ 7542.767151] LustreError: 191760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.277592] LustreError: 191742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949206 with bad export cookie 4603061543431456131 [ 7546.278280] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7546.284388] LustreError: 191742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7546.813277] Lustre: server umount lustre-MDT0001 complete [ 7561.578044] Lustre: server umount lustre-OST0000 complete [ 7570.984845] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7583.366687] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7616.654597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.953372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7616.981817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7617.051891] Lustre: lustre-MDT0000: new disk, initializing [ 7617.136824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.157658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7620.874749] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7629.589475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.673326] Lustre: 197153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7629.698257] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7629.700843] Lustre: Skipped 1 previous similar message [ 7629.702240] LustreError: 197935:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7629.706857] LustreError: 197935:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7629.710253] Lustre: 197935:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7629.729250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7629.734727] Lustre: Skipped 1 previous similar message [ 7629.795421] Lustre: lustre-MDT0001: new disk, initializing [ 7629.878208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7629.907759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7629.919185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7633.700647] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7644.950673] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7652.113406] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7658.975247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7659.400580] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7659.402398] Lustre: Skipped 1 previous similar message [ 7659.405172] LustreError: 199098:0:(llog.c:1696:llog_backup()) MGC192.168.202.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7659.418043] LustreError: 199098:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7659.423872] Lustre: 199098:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.149@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7659.430740] Lustre: 199098:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7659.456777] Lustre: lustre-OST0000: new disk, initializing [ 7659.459834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7659.465305] Lustre: 199123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7659.553302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.627979] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7666.801173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7666.809739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7666.964780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7676.004032] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7680.664626] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7680.999486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7685.145383] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7685.314402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7687.663591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7687.667079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7687.670040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7687.689377] Lustre: Skipped 1 previous similar message [ 7692.777911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7692.783991] Lustre: Skipped 2 previous similar messages [ 7692.999624] Lustre: server umount lustre-OST0000 complete [ 7697.890224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.896172] Lustre: Skipped 1 previous similar message [ 7703.015124] LustreError: 197160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7703.029319] LustreError: 197160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7703.208566] Lustre: server umount lustre-MDT0000 complete [ 7708.070885] LustreError: 197147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949368 with bad export cookie 4603061543431456852 [ 7708.096572] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7708.795843] Lustre: server umount lustre-MDT0001 complete [ 7717.838460] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:22:56 (1788949376) [ 7733.924543] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 7744.869603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.368516] LustreError: 202250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.444149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.983306] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7750.631359] LustreError: 202251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7758.132694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.904408] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7767.708524] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7777.087311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7777.495142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.505030] Lustre: Skipped 1 previous similar message [ 7785.017123] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7795.362918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.513345] Lustre: lustre-OST0001: new disk, initializing [ 7795.522922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7795.530582] Lustre: 204557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7802.791941] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7804.078114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7804.090334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7804.232473] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7814.037925] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7824.776664] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7827.007781] Lustre: DEBUG MARKER: ADD OST3 [ 7832.371462] loop: module loaded [ 7832.462460] loop0: detected capacity change from 0 to 204800000 [ 7832.691374] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7832.705885] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7832.717546] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7838.970885] loop0: detected capacity change from 0 to 204800000 [ 7839.976500] Lustre: DEBUG MARKER: STOP OST3 [ 7846.146751] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7846.512893] Lustre: lustre-OST0003: new disk, initializing [ 7846.517634] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7846.530985] Lustre: 206581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7846.703178] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7846.714923] Lustre: Skipped 1 previous similar message [ 7848.190842] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7848.328383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7848.344651] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7848.395659] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7850.209807] LustreError: 206732:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7850.431488] Lustre: Failing over lustre-OST0003 [ 7850.863199] Lustre: server umount lustre-OST0003 complete [ 7853.223468] LustreError: 203738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7853.253456] LustreError: 203738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7853.539868] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7853.557858] LustreError: Skipped 1 previous similar message [ 7853.566364] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.594052] Lustre: Skipped 1 previous similar message [ 7853.979293] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7863.210465] Lustre: DEBUG MARKER: START OST3 again [ 7870.080255] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7870.336737] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7871.911633] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7872.167250] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7872.169393] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7876.758401] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 7898.410119] Lustre: Failing over lustre-OST0003 [ 7898.506601] Lustre: server umount lustre-OST0003 complete [ 7901.181859] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.202356] LustreError: 208677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.220448] LustreError: 208677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7916.514423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.520397] Lustre: Skipped 2 previous similar messages [ 7921.772691] Lustre: server umount lustre-MDT0000 complete [ 7925.158289] LustreError: 202230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949585 with bad export cookie 4603061543431457734 [ 7925.161216] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7925.180535] LustreError: 202230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7926.756790] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.792432] Lustre: Skipped 5 previous similar messages [ 7931.780172] Lustre: server umount lustre-MDT0001 complete [ 7937.139571] Lustre: server umount lustre-OST0000 complete [ 7942.429577] Lustre: server umount lustre-OST0001 complete [ 7952.332814] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 7962.539560] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8003.752777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.041447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8004.112929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8004.290705] Lustre: lustre-MDT0000: new disk, initializing [ 8004.354039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8004.362510] Lustre: Skipped 1 previous similar message [ 8004.377777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8008.265174] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8018.226668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8018.386104] Lustre: 212020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8018.779033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8023.478490] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8032.875319] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8038.376837] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8045.001624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8045.442276] Lustre: lustre-OST0000: new disk, initializing [ 8045.451381] Lustre: Skipped 1 previous similar message [ 8045.460595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8045.470481] Lustre: Skipped 2 previous similar messages [ 8045.483268] Lustre: 213987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8046.821806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8046.827804] Lustre: Skipped 1 previous similar message [ 8046.835384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8046.957547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8051.740796] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8060.600458] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8065.521325] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8065.814095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8070.654738] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8071.053915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8077.795602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.798060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.800950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8077.816353] Lustre: Skipped 2 previous similar messages [ 8077.854246] Lustre: Skipped 12 previous similar messages [ 8079.520875] Lustre: server umount lustre-OST0000 complete [ 8085.472888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8085.488385] LustreError: Skipped 1 previous similar message [ 8090.056366] Lustre: server umount lustre-MDT0000 complete [ 8093.159553] LustreError: 213994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8093.174487] LustreError: 213994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8094.159200] LustreError: 212815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949754 with bad export cookie 4603061543431460030 [ 8094.172372] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8094.180341] LustreError: 212815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8094.509096] Lustre: server umount lustre-MDT0001 complete [ 8098.094747] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:29:16 (1788949756) [ 8114.357140] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8124.677407] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8161.670659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.031112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8162.111695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8162.269305] Lustre: lustre-MDT0000: new disk, initializing [ 8162.452526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8167.669802] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8178.295544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8178.384490] Lustre: 218602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8178.736598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8183.584294] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8193.184433] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8199.177686] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8206.156766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8206.468405] Lustre: 220569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8208.183415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8213.099973] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8223.331430] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8227.503390] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8227.655647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8233.103469] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8233.363931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8239.100912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.107187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8239.115828] Lustre: Skipped 2 previous similar messages [ 8239.117582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.117589] Lustre: Skipped 3 previous similar messages [ 8241.885737] Lustre: server umount lustre-OST0000 complete [ 8252.491823] Lustre: server umount lustre-MDT0000 complete [ 8254.434749] LustreError: 218613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.457924] LustreError: 218613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8256.086803] LustreError: 218594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788949916 with bad export cookie 4603061543431460912 [ 8256.088897] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8256.097549] LustreError: 218594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8256.406261] Lustre: server umount lustre-MDT0001 complete [ 8265.900651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.527136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8266.532942] Lustre: Skipped 5 previous similar messages [ 8271.995507] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8282.208108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.591624] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8297.736324] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8305.681876] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8313.324424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8320.438026] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8328.678564] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8351.732287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8351.733528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8351.747166] LustreError: Skipped 1 previous similar message [ 8351.753728] Lustre: Skipped 4 previous similar messages [ 8355.970540] Lustre: server umount lustre-OST0000 complete [ 8374.239167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8374.490554] Lustre: server umount lustre-MDT0000 complete [ 8378.595644] LustreError: 222767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950038 with bad export cookie 4603061543431461787 [ 8378.606819] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8378.620258] LustreError: 222767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8379.096479] Lustre: server umount lustre-MDT0001 complete [ 8468.782659] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 8471.691248] Key type lgssc unregistered [ 8472.054916] LNet: 226477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8472.066960] LNetError: 226477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8472.081668] LNet: Removed LNI 192.168.202.149@tcp [ 8472.813202] Key type .llcrypt unregistered [ 8472.814952] Key type ._llcrypt unregistered [ 8499.767272] Key type ._llcrypt registered [ 8499.772418] Key type .llcrypt registered [ 8499.881557] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8515.219633] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8516.400680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8516.430510] alg: No test for adler32 (adler32-zlib) [ 8517.515317] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 8517.744113] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 8519.415436] Key type lgssc registered [ 8520.711725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8555.174252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8555.187191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.501902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8556.564160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8556.723900] Lustre: lustre-MDT0000: new disk, initializing [ 8556.802628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.834723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8562.450867] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8572.017345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8572.199966] Lustre: 229834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8572.253840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8572.258632] Lustre: Skipped 1 previous similar message [ 8572.350585] Lustre: lustre-MDT0001: new disk, initializing [ 8572.568470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8572.616073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8572.639373] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8576.576866] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8588.256974] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8595.478408] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8602.937946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8603.185685] Lustre: lustre-OST0000: new disk, initializing [ 8603.191539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8603.201362] Lustre: 231801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8603.274575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8609.508871] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8609.881509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8609.896298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8610.001161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8619.776778] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8623.956106] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8624.131842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8627.683604] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8627.873412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8630.760280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8630.765218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8630.786861] LustreError: Skipped 1 previous similar message [ 8630.789375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8630.817691] Lustre: Skipped 1 previous similar message [ 8635.880522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8635.889883] Lustre: Skipped 2 previous similar messages [ 8641.013312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8641.016472] Lustre: Skipped 1 previous similar message [ 8644.063127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8644.282364] Lustre: server umount lustre-OST0000 complete [ 8649.184965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8649.194198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.207110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8651.253476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.636311] Lustre: server umount lustre-MDT0000 complete [ 8659.702285] LustreError: 229825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950319 with bad export cookie 4093913032117886204 [ 8659.711733] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8659.715932] LustreError: 229825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8660.100589] Lustre: server umount lustre-MDT0001 complete [ 8687.312874] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8699.653132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.215303] LustreError: 234757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.244037] LustreError: 234757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8700.301183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8705.078980] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8705.506331] LustreError: 234758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8710.630080] LustreError: 234757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.085550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8714.662698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8720.201850] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8723.814558] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8732.147651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8732.471097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8740.085532] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8749.619513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8749.890391] Lustre: lustre-OST0001: new disk, initializing [ 8749.893346] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8749.899090] Lustre: 237065:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8749.992362] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8757.145652] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8757.473971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8757.480567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8757.568600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8768.416149] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8776.409708] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8791.183290] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:40:49 (1788950449) [ 8793.582240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.588758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8793.612326] Lustre: Skipped 3 previous similar messages [ 8793.625550] Lustre: Skipped 5 previous similar messages [ 8799.793689] Lustre: server umount lustre-MDT0000 complete [ 8803.809095] LustreError: 235485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8803.831956] LustreError: 235485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8804.565053] LustreError: 234738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950464 with bad export cookie 4093913032117887086 [ 8804.571214] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8804.590070] LustreError: 234738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8805.113305] Lustre: server umount lustre-MDT0001 complete [ 8820.309274] Lustre: server umount lustre-OST0000 complete [ 8835.098237] Lustre: server umount lustre-OST0001 complete [ 8843.360098] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 8853.325276] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 8896.724396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.039225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8897.094592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8897.164465] Lustre: lustre-MDT0000: new disk, initializing [ 8897.273316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8897.285770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8902.662732] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8913.794295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8914.003692] Lustre: 241013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8914.148037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8914.153156] Lustre: Skipped 1 previous similar message [ 8914.257692] Lustre: lustre-MDT0001: new disk, initializing [ 8914.465429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8914.484378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8919.355132] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8928.080601] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8933.489290] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8941.364489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8941.726250] Lustre: lustre-OST0000: new disk, initializing [ 8941.729474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8941.733853] Lustre: 242984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8943.617196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8943.636364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8943.778421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8948.185958] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 8958.460980] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8963.886947] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8964.115937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8969.343948] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8969.731249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8974.306293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8974.314494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.316761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8974.333197] Lustre: Skipped 1 previous similar message [ 8974.349346] Lustre: Skipped 5 previous similar messages [ 8978.490755] Lustre: server umount lustre-OST0000 complete [ 8984.552801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8984.575510] Lustre: Skipped 1 previous similar message [ 8988.874860] Lustre: server umount lustre-MDT0000 complete [ 8989.673284] LustreError: 241026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.696487] LustreError: 241026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8993.542894] LustreError: 241007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950653 with bad export cookie 4093913032117889221 [ 8993.561188] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8994.264498] Lustre: server umount lustre-MDT0001 complete [ 9003.722260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9004.405985] LustreError: 245200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9004.444211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9004.447511] Lustre: Skipped 2 previous similar messages [ 9009.491221] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9019.895464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9026.463826] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9034.548347] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9040.683145] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9047.222280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9056.286151] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9065.645587] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9213.922187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9213.928056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.930951] LustreError: Skipped 1 previous similar message [ 9213.932454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9213.932460] Lustre: Skipped 2 previous similar messages [ 9213.996440] Lustre: Skipped 1 previous similar message [ 9216.652411] Lustre: server umount lustre-OST0000 complete [ 9226.493962] Lustre: server umount lustre-MDT0000 complete [ 9229.284840] LustreError: 245197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9229.300569] LustreError: 245197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9230.094357] LustreError: 245182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788950890 with bad export cookie 4093913032117890103 [ 9230.113513] LustreError: 245182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9230.123270] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9230.384701] Lustre: server umount lustre-MDT0001 complete [ 9333.541330] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [ 9336.417685] Key type lgssc unregistered [ 9336.822885] LNet: 248811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9336.827222] LNetError: 248811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9336.843805] LNet: Removed LNI 192.168.202.149@tcp [ 9337.742192] Key type .llcrypt unregistered [ 9337.747709] Key type ._llcrypt unregistered [ 9363.604994] Key type ._llcrypt registered [ 9363.607812] Key type .llcrypt registered [ 9363.717399] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9380.112654] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9380.767911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9380.840894] alg: No test for adler32 (adler32-zlib) [ 9382.014599] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [ 9382.220269] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [ 9383.888157] Key type lgssc registered [ 9384.937291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9439.717935] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9448.997448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9449.027737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9450.355296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9450.380844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9450.445312] Lustre: lustre-MDT0000: new disk, initializing [ 9450.570841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.593338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9455.225803] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9464.083771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9464.174131] Lustre: 252881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9464.229265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9464.234143] Lustre: Skipped 1 previous similar message [ 9464.291776] Lustre: lustre-MDT0001: new disk, initializing [ 9464.368366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9464.396376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9464.407044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9468.955992] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9473.301390] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9479.998609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.336978] Lustre: lustre-OST0000: new disk, initializing [ 9480.338974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9480.344616] Lustre: 254512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9480.473652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.689700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9483.698488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9483.837027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9487.101055] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9497.979270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9498.215426] Lustre: lustre-OST0001: new disk, initializing [ 9498.217893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9498.228548] Lustre: 255381:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9498.373454] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9503.706183] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9506.887665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9506.894630] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9507.015956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9514.892643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9518.904209] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9530.849639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9530.859111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.888300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9532.900553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9532.904654] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9532.912145] Lustre: Skipped 1 previous similar message [ 9532.933948] Lustre: Skipped 1 previous similar message [ 9536.325037] Lustre: server umount lustre-MDT0000 complete [ 9540.826059] LustreError: 256151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951200 with bad export cookie 4334606086498586107 [ 9540.832785] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9540.846764] LustreError: 256151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9541.102922] Lustre: server umount lustre-MDT0001 complete [ 9555.642548] Lustre: server umount lustre-OST0000 complete [ 9559.327117] Lustre: 250387:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788951203/real 1788951203] req@ffff9b10826ce300 x1875851127890816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788951219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9559.362238] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9559.369500] Lustre: Skipped 1 previous similar message [ 9559.641466] Lustre: server umount lustre-OST0001 complete [ 9573.577672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9583.578632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9591.874709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9601.104587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9619.887880] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9630.959032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.987579] Lustre: lustre-MDT0000: reset Object Index mappings [ 9631.422630] LustreError: 260395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9631.432387] LustreError: 260395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9631.499609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9635.749455] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9636.844477] LustreError: 260396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9640.940860] LustreError: 260395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9644.225103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.276422] Lustre: lustre-MDT0001: reset Object Index mappings [ 9644.912319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9650.499726] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9653.889166] Lustre: 261537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9660.613578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9661.065706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9667.078429] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9669.294587] LustreError: 262368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9674.730560] LustreError: 261892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9675.900986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9677.208371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9677.245319] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9682.745524] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9691.316785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9695.007497] Lustre: 263408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9701.344538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9701.352333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9701.373835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9701.382276] Lustre: Skipped 2 previous similar messages [ 9702.436223] Lustre: server umount lustre-MDT0000 complete [ 9702.891932] LustreError: 260395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.915474] LustreError: 260395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9705.815543] LustreError: 260376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951365 with bad export cookie 4334606086498592897 [ 9705.815875] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9705.828835] LustreError: 260376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9706.247196] Lustre: server umount lustre-MDT0001 complete [ 9720.564697] Lustre: server umount lustre-OST0000 complete [ 9734.732966] Lustre: server umount lustre-OST0001 complete [ 9754.223891] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:56:53 (1788951413) [ 9770.752629] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [ 9780.989775] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9817.855857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.065379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9818.087737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9818.166585] Lustre: lustre-MDT0000: new disk, initializing [ 9818.248874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9818.261494] Lustre: Skipped 1 previous similar message [ 9818.279151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9823.367646] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9834.822897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.934520] Lustre: 267662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9834.972715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9834.975881] Lustre: Skipped 1 previous similar message [ 9835.020973] Lustre: lustre-MDT0001: new disk, initializing [ 9835.139586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9835.163714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9840.266205] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9850.373419] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9856.750558] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9863.746588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9864.088350] Lustre: lustre-OST0000: new disk, initializing [ 9864.091468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9864.096933] Lustre: 269631:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9866.175785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9866.180744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9866.318652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9870.599781] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9880.335988] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9886.780689] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9887.196492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9893.518052] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9893.859870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9896.931904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9896.934615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9896.949299] Lustre: Skipped 3 previous similar messages [ 9896.953620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9896.975415] Lustre: Skipped 1 previous similar message [ 9902.052723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9902.063642] Lustre: Skipped 1 previous similar message [ 9910.239148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9910.521821] Lustre: server umount lustre-OST0000 complete [ 9916.898276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9916.902600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.909392] Lustre: Skipped 1 previous similar message [ 9916.919298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9916.921900] Lustre: Skipped 2 previous similar messages [ 9929.183351] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9929.629640] Lustre: server umount lustre-MDT0000 complete [ 9932.769387] LustreError: 267675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9932.785153] LustreError: 267675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9933.279565] LustreError: 267656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951593 with bad export cookie 4334606086498594990 [ 9933.290438] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9933.727485] Lustre: server umount lustre-MDT0001 complete [ 9952.426877] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [ 9963.213134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.657481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9963.665807] Lustre: Skipped 2 previous similar messages [ 9968.694328] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9969.122724] LustreError: 272572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.141020] LustreError: 272572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9977.105415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9982.006665] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [ 9985.311355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9992.543393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9999.918536] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10009.746148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10009.943523] Lustre: lustre-OST0001: new disk, initializing [10009.950047] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10009.956982] Lustre: 274889:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10011.508839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10011.519246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10011.626226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10016.190789] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10025.842987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10030.693487] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10036.319190] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10036.408350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10043.875691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10043.883189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.898079] Lustre: Skipped 1 previous similar message [10043.904752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10043.913952] Lustre: Skipped 6 previous similar messages [10049.594436] Lustre: server umount lustre-MDT0000 complete [10052.580717] LustreError: 275278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10052.602870] LustreError: 275278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10053.282123] LustreError: 272552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951713 with bad export cookie 4334606086498595872 [10053.291497] LustreError: 272552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10053.292409] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10053.837301] Lustre: server umount lustre-MDT0001 complete [10068.401449] Lustre: server umount lustre-OST0000 complete [10082.275886] Lustre: server umount lustre-OST0001 complete [10099.528490] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10110.485979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.975864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.981025] Lustre: Skipped 3 previous similar messages [10115.469284] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10124.696765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.693454] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10132.999578] Lustre: 279087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10140.023247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10146.892666] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10155.715275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10157.014968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10157.035095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10161.389983] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10168.126856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10171.459584] Lustre: 280967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10175.626606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10175.815172] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10184.275706] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:04:03 (1788951843) [10201.058354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10201.068697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10201.072725] Lustre: Skipped 6 previous similar messages [10201.085967] Lustre: Skipped 7 previous similar messages [10205.104927] Lustre: server umount lustre-MDT0000 complete [10206.185291] LustreError: 279743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10206.231537] LustreError: 279743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10209.034527] LustreError: 281677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788951869 with bad export cookie 4334606086498602403 [10209.045785] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10209.056199] LustreError: 281677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10215.535653] Lustre: server umount lustre-MDT0001 complete [10225.569977] Lustre: server umount lustre-OST0000 complete [10235.651481] Lustre: server umount lustre-OST0001 complete [10244.662433] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [10256.280611] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10304.766789] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10314.523695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10314.749104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10314.780393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10314.840974] Lustre: lustre-MDT0000: new disk, initializing [10314.904524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10319.218634] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10329.167798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10329.294768] Lustre: 285212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10329.624027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10334.165307] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10344.694684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10344.952391] Lustre: 286843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10346.692418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10351.243807] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10361.754830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10361.953515] Lustre: 287715:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10363.852253] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10363.868277] Lustre: Skipped 1 previous similar message [10364.083452] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10368.834629] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10377.794600] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10381.101066] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10381.108189] Lustre: Skipped 1 previous similar message [10393.572689] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:07:32 (1788952052) [10396.130409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10396.143217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.167229] Lustre: Skipped 2 previous similar messages [10396.174541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.177115] Lustre: Skipped 4 previous similar messages [10398.155795] Lustre: server umount lustre-MDT0000 complete [10402.301673] LustreError: 285202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788952062 with bad export cookie 4334606086498610054 [10402.310438] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10402.320814] LustreError: 285202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10402.826806] Lustre: server umount lustre-MDT0001 complete [10417.434567] Lustre: server umount lustre-OST0000 complete [10420.705742] Lustre: 250388:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788952064/real 1788952064] req@ffff9b108ec34700 x1875851128196864/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788952080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10421.475082] Lustre: server umount lustre-OST0001 complete [10430.653968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10431.134030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10431.140037] Lustre: Skipped 7 previous similar messages [10436.068404] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10444.751435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.591760] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10456.862149] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10462.500460] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10469.254755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10470.567985] LustreError: 291787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10470.585093] LustreError: 291787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10475.650294] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10483.148938] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10500.179038] loop1: detected capacity change from 0 to 204800000 [10500.472638] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10500.486113] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10500.497272] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10505.491146] loop1: detected capacity change from 0 to 204800000 [10511.282330] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10511.350528] LustreError: 289948:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs10574: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10511.360851] LustreError: 289948:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs10574 [10511.365299] LustreError: 289948:0:(mgs_handler.c:517:mgs_target_reg()) fs10574-MDT0001: Error checking target index: rc = -13 [10511.371951] LustreError: fs10574-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10511.378473] LustreError: 293675:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10511.385665] LustreError: 293675:0:(tgt_mount.c:2129:server_put_super()) no obd fs10574-MDT0001 [10511.393155] LustreError: 293675:0:(tgt_mount.c:116:server_deregister_mount()) fs10574-MDT0001 not registered [10511.431509] Lustre: server umount fs10574-MDT0001 complete [10511.434656] LustreError: 293675:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10519.998536] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10520.073293] Lustre: Setting parameter fs10574-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs10574-MDT0001 [10520.103342] Lustre: srv-fs10574-MDT0001: No data found on store. Initialize space. [10520.106428] Lustre: Skipped 4 previous similar messages [10520.155303] Lustre: fs10574-MDT0001: new disk, initializing [10520.157193] Lustre: Skipped 3 previous similar messages [10520.228128] Lustre: srv-fs10574-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10520.231158] Lustre: 294142:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10574-MDT0001-mdtlov: can't create new llog: rc = -115 [10520.246284] LustreError: 294142:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10574-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10524.464315] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10532.459810] Lustre: server umount fs10574-MDT0001 complete [10549.152467] loop1: detected capacity change from 0 to 204800000 [10553.848648] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10556.019817] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10556.434602] Lustre: 295710:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10574-MDT0001-mdtlov: can't create new llog: rc = -115 [10556.457119] LustreError: 295710:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10574-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10561.808514] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10566.601889] Lustre: server umount fs10574-MDT0001 complete [10576.031532] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [10578.403384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.424286] Lustre: Skipped 4 previous similar messages [10582.103068] Lustre: server umount lustre-MDT0000 complete [10590.751807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10590.975891] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10596.192543] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10596.343443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10604.557400] loop1: detected capacity change from 0 to 204800000 [10610.317502] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.724070] Lustre: 298156:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs10574-MDT0001-mdtlov: can't create new llog: rc = -115 [10610.740658] LustreError: 298156:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs10574-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10615.654156] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10620.905600] Lustre: server umount fs10574-MDT0001 complete [10644.964487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10644.967622] LustreError: Skipped 2 previous similar messages [10650.685383] Lustre: server umount lustre-OST0000 complete [10659.992025] Lustre: server umount lustre-MDT0000 complete [10664.574243] LustreError: 296202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788952324 with bad export cookie 4334606086498612735 [10664.581628] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10665.199585] Lustre: server umount lustre-MDT0001 complete [10680.743030] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [10683.766616] Key type lgssc unregistered [10684.103382] LNet: 300504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10684.116606] LNetError: 300504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10684.136807] LNet: Removed LNI 192.168.202.149@tcp [10685.119190] Key type .llcrypt unregistered [10685.122746] Key type ._llcrypt unregistered [10706.208558] Key type ._llcrypt registered [10706.214707] Key type .llcrypt registered [10706.398830] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [10707.716814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10707.831353] alg: No test for adler32 (adler32-zlib) [10709.072486] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [10709.380557] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [10711.119187] Key type lgssc registered [10712.454347] Lustre: Echo OBD driver; http://www.lustre.org/ [10720.323235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10720.345083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10722.067936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10726.684755] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10732.527493] LustreError: 301775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10732.580932] LustreError: 301775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10736.633949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10737.016619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10742.271146] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10750.774991] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10756.877895] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10764.333954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10764.701299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10767.804712] LustreError: 303607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10772.297711] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10772.971519] LustreError: 303606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10778.097687] LustreError: 303605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.945964] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10782.190087] LustreError: 303779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.224865] LustreError: 303779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10792.422485] LustreError: 303607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.447674] LustreError: 303607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10801.055296] loop1: detected capacity change from 0 to 204800000 [10801.207609] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10801.223509] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10801.240778] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10806.487802] loop1: detected capacity change from 0 to 204800000 [10809.465899] LustreError: 303606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10809.487614] LustreError: 303606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10813.005829] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10813.275406] Lustre: fs21695-OST0000: new disk, initializing [10813.287343] Lustre: srv-fs21695-OST0000: No data found on store. Initialize space. [10813.311351] Lustre: 305511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10813.429422] Lustre: fs21695-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10821.307835] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10831.812894] Lustre: server umount fs21695-OST0000 complete [10841.813453] loop1: detected capacity change from 0 to 204800000 [10843.638652] LustreError: 303779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.657627] LustreError: 303779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [10843.672278] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10847.278570] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10847.625239] Lustre: fs21695-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10854.538351] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10863.900327] loop2: detected capacity change from 0 to 204800000 [10864.109147] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10864.114972] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10864.127558] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10864.128266] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10870.017505] loop2: detected capacity change from 0 to 204800000 [10876.688403] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10876.831277] LustreError: fs21695-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10876.841536] LustreError: fs21695-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10876.854568] LustreError: 308553:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10876.860121] LustreError: 308553:0:(tgt_mount.c:2129:server_put_super()) no obd fs21695-OST0001 [10876.869156] LustreError: 308553:0:(tgt_mount.c:116:server_deregister_mount()) fs21695-OST0001 not registered [10876.921626] Lustre: server umount fs21695-OST0001 complete [10876.930148] LustreError: 308553:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10889.511296] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.956654] Lustre: fs21695-OST0001: new disk, initializing [10889.972861] Lustre: srv-fs21695-OST0001: No data found on store. Initialize space. [10889.981974] Lustre: 309105:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10890.040363] Lustre: fs21695-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10897.287791] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10906.225913] Lustre: server umount fs21695-OST0000 complete [10910.194938] LustreError: 303605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10910.221489] LustreError: 303605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10916.707595] Lustre: server umount fs21695-OST0001 complete [10932.846129] loop1: detected capacity change from 0 to 204800000 [10943.469989] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10944.296254] Lustre: fs21695-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10954.193843] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10963.176110] loop2: detected capacity change from 0 to 204800000 [10971.763466] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10979.164052] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [10988.833980] Lustre: server umount fs21695-OST0000 complete [10997.229194] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10999.371658] Lustre: server umount fs21695-OST0001 complete [11011.555069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11011.564123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11011.573987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11012.577344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11012.594543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11017.566422] Lustre: server umount lustre-OST0000 complete [11022.817182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11022.842159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.854073] Lustre: Skipped 1 previous similar message [11027.936575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11027.945538] Lustre: Skipped 1 previous similar message [11028.924277] Lustre: server umount lustre-MDT0000 complete [11033.802981] LustreError: 309104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788952693 with bad export cookie 10339506266379461774 [11033.805391] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11033.820606] LustreError: 309104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11034.941882] Lustre: server umount lustre-MDT0001 complete [11047.518548] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [11050.938669] Key type lgssc unregistered [11051.292769] LNet: 315035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11051.301095] LNetError: 315035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11051.311143] LNet: Removed LNI 192.168.202.149@tcp [11052.221380] Key type .llcrypt unregistered [11052.223211] Key type ._llcrypt unregistered [11084.864511] Key type ._llcrypt registered [11084.866274] Key type .llcrypt registered [11084.969747] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [11098.933247] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11100.244033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11100.276946] alg: No test for adler32 (adler32-zlib) [11101.409646] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [11101.577681] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [11103.224887] Key type lgssc registered [11104.602935] Lustre: Echo OBD driver; http://www.lustre.org/ [11150.223174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11150.244406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11151.533280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11151.560098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11151.612173] Lustre: lustre-MDT0000: new disk, initializing [11151.667316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11151.689294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11156.216831] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11165.574865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.785060] Lustre: 318899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11165.902685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11165.918614] Lustre: Skipped 1 previous similar message [11166.065544] Lustre: lustre-MDT0001: new disk, initializing [11166.195300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11166.247168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11166.264746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11171.808080] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11185.988230] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11194.769557] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11203.733984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11204.335882] Lustre: lustre-OST0000: new disk, initializing [11204.354575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11204.382902] Lustre: 320870:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11204.692234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11211.481227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11211.501630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11211.814573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11212.965583] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11223.989724] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11226.213486] Lustre: Mounted lustre-client - version 2.17.57_55_ga2ebe9c [11227.171601] Lustre: Unmounted lustre-client [11237.349031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11237.357712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.361117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11237.369182] Lustre: Skipped 1 previous similar message [11237.378992] Lustre: Skipped 1 previous similar message [11241.441926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11241.444736] Lustre: Skipped 1 previous similar message [11243.300986] Lustre: server umount lustre-OST0000 complete [11248.099668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11248.119470] LustreError: Skipped 1 previous similar message [11248.131248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11248.168991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11251.681985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11251.694569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11251.731075] Lustre: Skipped 1 previous similar message [11253.925885] Lustre: server umount lustre-MDT0000 complete [11258.317265] LustreError: 319693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788952918 with bad export cookie 14107142497869677651 [11258.325896] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11258.330105] LustreError: 319693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11258.870152] Lustre: server umount lustre-MDT0001 complete [11272.906840] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [11275.640287] Key type lgssc unregistered [11284.321220] LNet: 323035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11284.338637] LNetError: 323035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11356.076267] LNet: Removed LNI 192.168.202.149@tcp [11357.148678] Key type .llcrypt unregistered [11357.150977] Key type ._llcrypt unregistered [11377.420152] Key type ._llcrypt registered [11377.422574] Key type .llcrypt registered [11377.608909] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [11395.139734] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11396.431983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11396.455100] alg: No test for adler32 (adler32-zlib) [11397.605554] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [11397.957426] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [11399.743281] Key type lgssc registered [11400.806141] Lustre: Echo OBD driver; http://www.lustre.org/ [11438.716694] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:24:57 (1788953097) [11446.257303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11446.309269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.557988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11447.583343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11447.647604] Lustre: lustre-MDT0000: new disk, initializing [11447.755376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11447.788477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11452.616915] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11463.850228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11464.033817] Lustre: 326258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11464.106331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11464.116906] Lustre: Skipped 1 previous similar message [11464.230068] Lustre: lustre-MDT0001: new disk, initializing [11464.381161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11464.423918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11464.443399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11469.263849] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11478.225415] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11484.811105] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11492.243410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11492.656358] Lustre: lustre-OST0000: new disk, initializing [11492.658304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11492.662293] Lustre: 328227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11492.767407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11499.539702] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11502.150381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11502.164159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11502.274266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11509.415370] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11528.160589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11528.169295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11528.185386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11528.193277] Lustre: Skipped 1 previous similar message [11528.211178] Lustre: Skipped 1 previous similar message [11529.976501] Lustre: server umount lustre-OST0000 complete [11536.353908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11536.364450] LustreError: Skipped 1 previous similar message [11536.370526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11536.396480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11538.400180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11538.400885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11543.530049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11543.537146] Lustre: Skipped 2 previous similar messages [11548.127149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11548.591520] Lustre: server umount lustre-MDT0000 complete [11552.350521] LustreError: 326248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788953212 with bad export cookie 16694363725945958448 [11552.356030] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11552.365191] LustreError: 326248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11552.789181] Lustre: server umount lustre-MDT0001 complete [11564.653168] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [11567.825171] Key type lgssc unregistered [11568.119572] LNet: 330140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11568.124862] LNetError: 330140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11568.141715] LNet: Removed LNI 192.168.202.149@tcp [11568.993566] Key type .llcrypt unregistered [11568.996710] Key type ._llcrypt unregistered [11589.459561] Key type ._llcrypt registered [11589.462521] Key type .llcrypt registered [11589.629411] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11591.384641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11591.395508] alg: No test for adler32 (adler32-zlib) [11592.631734] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [11592.980872] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [11594.640131] Key type lgssc registered [11595.898397] Lustre: Echo OBD driver; http://www.lustre.org/ [11604.395209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11604.407582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.069557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11611.326742] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11616.225348] LustreError: 331416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11616.252707] LustreError: 331416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11621.359540] LustreError: 331415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11621.421438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11621.874470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11626.964897] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11635.419848] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11641.472407] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11648.500630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11648.974351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11655.559653] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11664.120201] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11674.137493] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11675.726440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11681.765454] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11681.774621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11681.796663] Lustre: Skipped 1 previous similar message [11681.804157] Lustre: Skipped 1 previous similar message [11682.490052] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.49@tcp (stopping) [11682.514067] Lustre: Skipped 1 previous similar message [11686.891770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11686.904908] Lustre: Skipped 2 previous similar messages [11687.540555] LustreError: 331412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11687.842892] Lustre: server umount lustre-MDT0000 complete [11692.006304] LustreError: 333827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.468663] LustreError: 331396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788953354 with bad export cookie 2261630410842928041 [11694.482958] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11695.094661] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11695.132473] Lustre: Skipped 1 previous similar message [11695.136229] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11697.783784] LustreError: 333827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.812847] LustreError: 333827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11700.803267] Lustre: server umount lustre-MDT0001 complete [11705.273593] Lustre: server umount lustre-OST0000 complete [11715.488151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.111843] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11724.672935] Lustre: 335597:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.149@tcp [11727.190882] Lustre: server umount MGS complete [11737.512838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.720995] LustreError: 336167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.728884] LustreError: 336167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11738.162656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11744.036980] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11753.001856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11753.415628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11759.027546] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11769.563803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11770.100647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11777.971183] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11785.774099] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11787.743926] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11793.494001] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11795.219362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11801.062206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11801.067872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11801.079688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11801.081441] Lustre: Skipped 2 previous similar messages [11801.548892] Lustre: server umount lustre-OST0000 complete [11809.260193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.265393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11809.276065] Lustre: Skipped 1 previous similar message [11809.292558] Lustre: Skipped 1 previous similar message [11811.836994] Lustre: server umount lustre-MDT0000 complete [11814.372716] LustreError: 337346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11814.396846] LustreError: 337346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11815.372884] LustreError: 337628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788953475 with bad export cookie 2261630410842928881 [11815.390014] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11815.396941] LustreError: 337628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11825.843887] Lustre: server umount lustre-MDT0001 complete [11838.785760] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [11842.671583] Key type lgssc unregistered [11843.005414] LNet: 339610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11843.018149] LNetError: 339610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11843.052979] LNet: Removed LNI 192.168.202.149@tcp [11844.358995] Key type .llcrypt unregistered [11844.362147] Key type ._llcrypt unregistered [11864.399739] Key type ._llcrypt registered [11864.401554] Key type .llcrypt registered [11864.477592] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [11885.636301] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [11887.878659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11887.940356] alg: No test for adler32 (adler32-zlib) [11889.219356] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [11889.618419] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [11891.391248] Key type lgssc registered [11893.535200] Lustre: Echo OBD driver; http://www.lustre.org/ [11939.350863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11939.373301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11940.601453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11940.632774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11940.712599] Lustre: lustre-MDT0000: new disk, initializing [11940.760626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11940.771885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11946.258254] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11958.035576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.144028] Lustre: 342772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11958.194443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11958.197868] Lustre: Skipped 1 previous similar message [11958.269781] Lustre: lustre-MDT0001: new disk, initializing [11958.423755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11958.490681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11958.504272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11964.161264] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [11974.681976] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11981.900662] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11990.297470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11990.778217] Lustre: lustre-OST0000: new disk, initializing [11990.798744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11990.809541] Lustre: 344743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11990.987959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11997.803243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11997.821311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11997.953241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11999.740767] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12011.612288] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12017.780594] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12017.958858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12023.439048] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12023.730807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12028.919032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12028.921614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12028.938648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12028.943482] LustreError: Skipped 1 previous similar message [12028.989837] Lustre: Skipped 1 previous similar message [12034.021724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12034.035187] Lustre: Skipped 2 previous similar messages [12039.140611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12040.159798] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12040.301466] Lustre: server umount lustre-OST0000 complete [12045.280703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12045.293068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12045.334850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12045.341169] Lustre: Skipped 1 previous similar message [12049.379581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12049.393868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12051.494447] Lustre: server umount lustre-MDT0000 complete [12055.556950] LustreError: 342765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788953715 with bad export cookie 14202547024024906105 [12055.574804] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12056.214515] Lustre: server umount lustre-MDT0001 complete [12065.846412] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:35:24 (1788953724) [12084.897408] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [12097.672488] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [12138.645374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12139.083149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12139.150718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12139.274120] Lustre: lustre-MDT0000: new disk, initializing [12139.397298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12139.411800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12144.803562] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12156.616597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12156.893852] Lustre: 349504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12156.963846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12156.968892] Lustre: Skipped 1 previous similar message [12157.084979] Lustre: lustre-MDT0001: new disk, initializing [12157.216121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12157.257199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12157.271176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12163.196400] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12174.634370] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12180.835805] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12189.002032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12189.335452] Lustre: lustre-OST0000: new disk, initializing [12189.339401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12189.347083] Lustre: 351475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12189.457210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12197.541259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12197.560964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12197.574664] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12197.678363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12209.942419] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12249.057737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12249.058667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12249.070244] Lustre: Skipped 1 previous similar message [12249.096270] Lustre: Skipped 2 previous similar messages [12254.975241] Lustre: server umount lustre-OST0000 complete [12259.296727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.309753] Lustre: Skipped 1 previous similar message [12265.621677] Lustre: server umount lustre-MDT0000 complete [12269.537543] LustreError: 349512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.563766] LustreError: 349512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12270.026846] LustreError: 349496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788953930 with bad export cookie 14202547024024906987 [12270.035039] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12270.038230] LustreError: 349496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12270.585200] Lustre: server umount lustre-MDT0001 complete [12283.593603] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12286.108708] Key type lgssc unregistered [12286.396856] LNet: 353653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12286.401469] LNetError: 353653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12286.416074] LNet: Removed LNI 192.168.202.149@tcp [12287.074757] Key type .llcrypt unregistered [12287.077991] Key type ._llcrypt unregistered [12311.712496] Key type ._llcrypt registered [12311.714932] Key type .llcrypt registered [12311.866107] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [12313.701757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12313.810801] alg: No test for adler32 (adler32-zlib) [12314.958718] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [12315.240813] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [12316.943740] Key type lgssc registered [12318.737450] Lustre: Echo OBD driver; http://www.lustre.org/ [12328.253371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12328.276341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12329.906352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12335.716492] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12340.198540] LustreError: 355026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12340.218113] LustreError: 355026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12345.322954] LustreError: 355025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12345.432116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12345.782628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12351.365530] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12359.680355] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12365.845978] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12374.111808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12374.485942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12380.996249] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12390.018190] Lustre: DEBUG MARKER: oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12411.380694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12411.394386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12411.414668] Lustre: Skipped 2 previous similar messages [12411.417376] Lustre: Skipped 1 previous similar message [12415.435470] Lustre: server umount lustre-MDT0000 complete [12416.484678] LustreError: 355025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12416.507106] LustreError: 355025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12419.179121] LustreError: 355006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788954079 with bad export cookie 5247083904646980552 [12419.184198] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12419.187724] LustreError: 355006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12419.376702] Lustre: server umount lustre-MDT0001 complete [12432.690716] Lustre: server umount lustre-OST0000 complete [12477.699526] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [12489.793159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.249085] LustreError: 360254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.308810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12494.841026] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12495.333727] LustreError: 360255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.889537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12504.467918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12509.693296] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12513.089604] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12520.801859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12521.302387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12528.485707] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12535.673215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [12538.083431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12538.393682] Lustre: lustre-OST0001: new disk, initializing [12538.396856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12538.402320] Lustre: 362562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12538.553358] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12545.120174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12545.126872] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12545.248729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12545.762213] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12555.652626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12562.938498] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12565.534737] Lustre: DEBUG MARKER: == conf-sanity test 301: lfind --fid2path names a device scan's objects ========================================================== 07:43:43 (1788954223) [12570.085785] LustreError: 354335:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [12570.117049] Lustre: Mounted lustre-client - version 2.17.57_55_ga2ebe9c [12573.854912] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12588.439311] Lustre: Unmounted lustre-client [12591.175518] Lustre: DEBUG MARKER: == conf-sanity test 302: lfind --fid2path names objects on a stopped OST ========================================================== 07:44:09 (1788954249) [12596.579684] Lustre: Mounted lustre-client - version 2.17.57_55_ga2ebe9c [12600.755379] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12605.620481] Lustre: Failing over lustre-OST0000 [12606.436172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12606.453006] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12606.478338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12606.483225] Lustre: Skipped 1 previous similar message [12607.748429] Lustre: server umount lustre-OST0000 complete [12611.709140] LustreError: 361737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.736761] LustreError: 361737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12625.311964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12625.544300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12625.560134] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12626.622059] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 4 clients reconnect [12627.009577] Lustre: lustre-OST0000: Recovery over after 0:01, of 4 clients 4 recovered and 0 were evicted. [12627.010497] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [12632.006093] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [12638.727113] Lustre: Unmounted lustre-client [12641.801807] Lustre: DEBUG MARKER: == conf-sanity test 303: lfind --paths names an MDT with no mount at all ========================================================== 07:45:00 (1788954300) [12661.988554] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 07:45:20 (1788954320) [12668.384664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12668.398153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12668.407059] Lustre: Skipped 2 previous similar messages [12668.411676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12668.416642] Lustre: Skipped 3 previous similar messages [12671.971553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12671.984100] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12672.016530] Lustre: Skipped 2 previous similar messages [12674.380775] Lustre: server umount lustre-MDT0000 complete [12677.108706] LustreError: 360249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12677.129036] LustreError: 360249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [12678.959293] LustreError: 360235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788954339 with bad export cookie 5247083904646985186 [12678.960792] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12678.965019] LustreError: 360235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12679.433544] Lustre: server umount lustre-MDT0001 complete [12694.386777] Lustre: server umount lustre-OST0000 complete [12698.655113] Lustre: 354337:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788954342/real 1788954342] req@ffff9b10915ab480 x1875854203472000/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788954358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12698.689394] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12698.948251] Lustre: server umount lustre-OST0001 complete [12743.139803] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 07:46:41 (1788954401) [12787.916153] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12791.549933] Key type lgssc unregistered [12792.038795] LNet: 369821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12792.048270] LNetError: 369821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12793.062370] LNet: Removed LNI 192.168.202.149@tcp [12794.240393] Key type .llcrypt unregistered [12794.242088] Key type ._llcrypt unregistered [12868.597108] Key type ._llcrypt registered [12868.607912] Key type .llcrypt registered [12868.764729] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12869.247734] Key type .llcrypt unregistered [12869.254776] Key type ._llcrypt unregistered [12905.337863] Key type ._llcrypt registered [12905.343297] Key type .llcrypt registered [12905.482779] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12905.900288] Key type .llcrypt unregistered [12905.901625] Key type ._llcrypt unregistered [12936.551672] Key type ._llcrypt registered [12936.553573] Key type .llcrypt registered [12936.667666] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [12937.056988] Key type .llcrypt unregistered [12937.064321] Key type ._llcrypt unregistered [13018.734837] Key type ._llcrypt registered [13018.736928] Key type .llcrypt registered [13018.882798] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13019.178793] Key type .llcrypt unregistered [13019.183185] Key type ._llcrypt unregistered [13052.088630] Key type ._llcrypt registered [13052.093227] Key type .llcrypt registered [13052.291686] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13052.735472] Key type .llcrypt unregistered [13052.742606] Key type ._llcrypt unregistered [13084.416911] Key type ._llcrypt registered [13084.418865] Key type .llcrypt registered [13084.573207] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13085.157973] Key type .llcrypt unregistered [13085.166405] Key type ._llcrypt unregistered [13124.419458] Key type ._llcrypt registered [13124.424568] Key type .llcrypt registered [13124.546476] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13124.917705] Key type .llcrypt unregistered [13124.921743] Key type ._llcrypt unregistered [13159.309648] Key type ._llcrypt registered [13159.312500] Key type .llcrypt registered [13159.444711] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13159.781423] Key type .llcrypt unregistered [13159.783976] Key type ._llcrypt unregistered [13197.518807] Key type ._llcrypt registered [13197.522190] Key type .llcrypt registered [13197.650234] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13198.013415] Key type .llcrypt unregistered [13198.017517] Key type ._llcrypt unregistered [13275.143447] Key type ._llcrypt registered [13275.147068] Key type .llcrypt registered [13275.296522] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13275.700954] Key type .llcrypt unregistered [13275.708535] Key type ._llcrypt unregistered [13310.778715] Key type ._llcrypt registered [13310.780891] Key type .llcrypt registered [13310.863855] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13311.295647] Key type .llcrypt unregistered [13311.305738] Key type ._llcrypt unregistered [13349.322911] Key type ._llcrypt registered [13349.324795] Key type .llcrypt registered [13349.444824] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13349.730280] Key type .llcrypt unregistered [13349.731492] Key type ._llcrypt unregistered [13413.237247] Key type ._llcrypt registered [13413.243406] Key type .llcrypt registered [13413.383485] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [13431.208738] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13433.109894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13433.203356] alg: No test for adler32 (adler32-zlib) [13434.334436] Lustre: Lustre: Build Version: 2.17.57_55_ga2ebe9c [13434.706140] LNet: Added LNI 192.168.202.149@tcp [8/256/0/180] [13436.431191] Key type lgssc registered [13438.004296] Lustre: Echo OBD driver; http://www.lustre.org/ [13472.452761] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13472.505988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13473.868686] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13473.928536] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13474.091915] Lustre: tst250fs-MDT0000: new disk, initializing [13474.166373] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13474.185880] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13479.017862] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13489.733164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13489.878236] Lustre: 387373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13489.948481] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13489.958379] Lustre: Skipped 1 previous similar message [13490.039972] Lustre: tst250fs-MDT0001: new disk, initializing [13490.120087] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13490.164220] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13490.187389] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13495.474454] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13500.503902] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13506.860845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13507.181613] Lustre: tst250fs-OST0000: new disk, initializing [13507.192913] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13507.210606] Lustre: 389003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13507.341948] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13513.888863] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13516.413117] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13516.423577] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13516.618056] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13525.929108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13526.110488] Lustre: tst250fs-OST0001: new disk, initializing [13526.113383] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13526.118326] Lustre: 389876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13526.211562] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13532.794467] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_default_debug -1 all [13533.374339] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13533.388093] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13533.499601] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13538.874404] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13541.767819] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13541.772446] Lustre: Skipped 2 previous similar messages [13559.268731] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13559.275895] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13559.301271] Lustre: Skipped 3 previous similar messages [13559.317176] Lustre: Skipped 3 previous similar messages [13564.390794] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13564.414281] Lustre: Skipped 3 previous similar messages [13565.229913] Lustre: server umount tst250fs-MDT0000 complete [13568.820817] LustreError: 387364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788955228 with bad export cookie 10210472813738199024 [13568.824948] LustreError: MGC192.168.202.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13568.827703] LustreError: 387364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13569.146259] Lustre: server umount tst250fs-MDT0001 complete [13583.569697] Lustre: server umount tst250fs-OST0000 complete [13585.695106] Lustre: 385600:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788955229/real 1788955229] req@ffff9b10811bbb80 x1875855376716672/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788955245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13585.726113] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13586.777203] Lustre: server umount tst250fs-OST0001 complete [13609.314653] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [13618.992585] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13655.689362] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:01:53 (1788955313) [13657.709880] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13675.236353] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing set_hostid [13687.063257] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing load_modules_local [13718.378094] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13424 sec ========= 08:02:56 (1788955376) [13720.639439] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:02:59 (1788955379) === [13723.725245] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:03:02 (1788955382) === [13770.487182] Lustre: DEBUG MARKER: oleg249-server.virtnet: executing unload_modules_local [13773.551910] Key type lgssc unregistered [13773.955441] LNet: 399035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13773.964817] LNetError: 399035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13773.980832] LNet: Removed LNI 192.168.202.149@tcp [13775.024170] Key type .llcrypt unregistered [13775.028654] Key type ._llcrypt unregistered