[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 528117106 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003135] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011062] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.022679] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027182] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031144] *** VALIDATE tmpfs *** [ 0.033008] *** VALIDATE proc *** [ 0.034250] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040004] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044555] debug: unmapping init [mem 0xffffffffb8059000-0xffffffffb8060fff] [ 0.046191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049020] ... bit width: 48 [ 0.050018] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053030] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055294] rcu: Hierarchical SRCU implementation. [ 0.057495] smp: Bringing up secondary CPUs ... [ 0.058619] x86: Booting SMP configuration: [ 0.059027] .... node #0, CPUs: #1 #2 #3 [ 0.062401] smp: Brought up 1 node, 4 CPUs [ 0.063982] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151070] node 0 deferred pages initialised in 85ms [ 0.156037] devtmpfs: initialized [ 0.157347] x86/mm: Memory block size: 128MB [ 0.160880] gcov: version magic: 0x41383552 [ 0.164296] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165139] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168412] pinctrl core: initialized pinctrl subsystem [ 0.170239] [ 0.170901] ************************************************************* [ 0.173021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176013] ** ** [ 0.178015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180013] ** ** [ 0.181012] ** This means that this kernel is built to expose internal ** [ 0.184018] ** IOMMU data structures, which may compromise security on ** [ 0.186014] ** your system. ** [ 0.189018] ** ** [ 0.191014] ** If you see this message and you are not debugging the ** [ 0.194014] ** kernel, report this immediately to your vendor! ** [ 0.196017] ** ** [ 0.199017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.201011] ************************************************************* [ 0.204970] NET: Registered protocol family 16 [ 0.207443] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218044] cpuidle: using governor menu [ 0.219835] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.223486] PCI: Using configuration type 1 for base access [ 0.226140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.240024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.244126] cryptd: max_cpu_qlen set to 1000 [ 0.247244] ACPI: Added _OSI(Module Device) [ 0.248014] ACPI: Added _OSI(Processor Device) [ 0.249014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255989] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262342] ACPI: Interpreter enabled [ 0.264057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.266017] ACPI: Using IOAPIC for interrupt routing [ 0.267122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.271424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.282402] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.287025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.290079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295428] acpiphp: Slot [2] registered [ 0.297127] acpiphp: Slot [5] registered [ 0.298121] acpiphp: Slot [6] registered [ 0.299116] acpiphp: Slot [7] registered [ 0.300107] acpiphp: Slot [8] registered [ 0.302131] acpiphp: Slot [9] registered [ 0.303122] acpiphp: Slot [10] registered [ 0.305108] acpiphp: Slot [3] registered [ 0.306104] acpiphp: Slot [4] registered [ 0.308108] acpiphp: Slot [11] registered [ 0.309112] acpiphp: Slot [12] registered [ 0.311120] acpiphp: Slot [13] registered [ 0.312099] acpiphp: Slot [14] registered [ 0.314124] acpiphp: Slot [15] registered [ 0.315120] acpiphp: Slot [16] registered [ 0.317100] acpiphp: Slot [17] registered [ 0.318104] acpiphp: Slot [18] registered [ 0.320093] acpiphp: Slot [19] registered [ 0.321122] acpiphp: Slot [20] registered [ 0.323098] acpiphp: Slot [21] registered [ 0.324107] acpiphp: Slot [22] registered [ 0.326108] acpiphp: Slot [23] registered [ 0.327114] acpiphp: Slot [24] registered [ 0.328080] acpiphp: Slot [25] registered [ 0.329130] acpiphp: Slot [26] registered [ 0.331177] acpiphp: Slot [27] registered [ 0.332096] acpiphp: Slot [28] registered [ 0.334116] acpiphp: Slot [29] registered [ 0.335114] acpiphp: Slot [30] registered [ 0.337129] acpiphp: Slot [31] registered [ 0.338048] PCI host bridge to bus 0000:00 [ 0.339018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354217] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357229] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360334] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370765] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.378047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.383024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.386578] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.390899] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397743] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.416972] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.420016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.427040] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.453030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.463995] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.475020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.503016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514751] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.519022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.525021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.538016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545358] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.558017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.579119] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.584033] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.592019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.606024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.615686] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.622020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.629022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.648023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.661049] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.663388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.665281] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.666307] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.669374] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.675012] iommu: Default domain type: Passthrough [ 0.676553] SCSI subsystem initialized [ 0.678152] ACPI: bus type USB registered [ 0.679106] usbcore: registered new interface driver usbfs [ 0.680091] usbcore: registered new interface driver hub [ 0.682096] usbcore: registered new device driver usb [ 0.684199] pps_core: LinuxPPS API ver. 1 registered [ 0.686013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.690065] PTP clock support registered [ 0.692077] EDAC MC: Ver: 3.0.0 [ 0.694123] PCI: Using ACPI for IRQ routing [ 0.695625] NetLabel: Initializing [ 0.697009] NetLabel: domain hash size = 128 [ 0.698009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.699060] NetLabel: unlabeled traffic allowed by default [ 0.701138] vgaarb: loaded [ 0.702208] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.708315] clocksource: Switched to clocksource kvm-clock [ 0.808675] VFS: Disk quotas dquot_6.6.0 [ 0.809939] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.812436] *** VALIDATE ramfs *** [ 0.813707] *** VALIDATE hugetlbfs *** [ 0.814959] pnp: PnP ACPI init [ 0.817931] pnp: PnP ACPI: found 6 devices [ 0.835111] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838692] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.840790] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.842455] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.844994] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.847844] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.850879] NET: Registered protocol family 2 [ 0.853768] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.858961] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.862141] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.867864] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.871789] TCP: Hash tables configured (established 65536 bind 65536) [ 0.875444] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.878747] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881746] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884750] NET: Registered protocol family 1 [ 0.889350] RPC: Registered named UNIX socket transport module. [ 0.891693] RPC: Registered udp transport module. [ 0.893441] RPC: Registered tcp transport module. [ 0.895411] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.897904] NET: Registered protocol family 44 [ 0.899766] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.902161] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.904372] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.907115] PCI: CLS 0 bytes, default 64 [ 0.909146] Unpacking initramfs... [ 2.323082] debug: unmapping init [mem 0xffff959ffcc54000-0xffff959ffffbffff] [ 2.326940] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.329148] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.331815] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.902110] Initialise system trusted keyrings [ 2.903917] Key type blacklist registered [ 2.905942] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.915798] zbud: loaded [ 2.919434] *** VALIDATE nfs *** [ 2.920860] *** VALIDATE nfs4 *** [ 2.922932] pstore: using deflate compression [ 2.926753] Platform Keyring initialized [ 3.030191] NET: Registered protocol family 38 [ 3.031521] Key type asymmetric registered [ 3.032566] Asymmetric key parser 'x509' registered [ 3.033811] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.036377] io scheduler mq-deadline registered [ 3.038051] io scheduler kyber registered [ 3.039713] io scheduler bfq registered [ 3.041862] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.044843] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.047814] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.050710] ACPI: Power Button [PWRF] [ 3.055645] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.060296] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.071059] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.075173] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.084333] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.108534] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.135119] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.140845] Non-volatile memory driver v1.3 [ 3.142222] Linux agpgart interface v0.103 [ 3.170354] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.173396] vda: detected capacity change from 0 to 74698752 [ 3.189652] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.192179] vdb: detected capacity change from 0 to 1073741824 [ 3.203656] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.205346] vdc: detected capacity change from 0 to 2621440000 [ 3.219337] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.221906] vdd: detected capacity change from 0 to 2621440000 [ 3.239643] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.242622] vde: detected capacity change from 0 to 4294967296 [ 3.260950] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.263592] vdf: detected capacity change from 0 to 4294967296 [ 3.280372] libphy: Fixed MDIO Bus: probed [ 3.294213] usbcore: registered new interface driver usbserial_generic [ 3.295747] usbserial: USB Serial support registered for generic [ 3.298313] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.302845] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.304802] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.307640] mousedev: PS/2 mouse device common for all mice [ 3.312356] rtc_cmos 00:05: RTC can wake from S4 [ 3.315563] rtc_cmos 00:05: registered as rtc0 [ 3.317817] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.320914] intel_pstate: CPU model not supported [ 3.324105] hid: raw HID events driver (C) Jiri Kosina [ 3.324268] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.326016] usbcore: registered new interface driver usbhid [ 3.326021] usbhid: USB HID core driver [ 3.326139] drop_monitor: Initializing network drop monitor service [ 3.326255] Initializing XFRM netlink socket [ 3.326585] NET: Registered protocol family 10 [ 3.327903] Segment Routing with IPv6 [ 3.333952] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.335373] NET: Registered protocol family 17 [ 3.339669] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.340489] mpls_gso: MPLS GSO support [ 3.353558] RAS: Correctable Errors collector initialized. [ 3.354819] AVX version of gcm_enc/dec engaged. [ 3.355756] AES CTR mode by8 optimization enabled [ 3.419307] sched_clock: Marking stable (3419285665, 0)->(4286509906, -867224241) [ 3.421905] registered taskstats version 1 [ 3.423313] Loading compiled-in X.509 certificates [ 3.424737] zswap: loaded using pool lzo/zbud [ 3.447595] Key type big_key registered [ 3.458843] Key type encrypted registered [ 3.460764] ima: No TPM chip found, activating TPM-bypass! [ 3.463093] ima: Allocated hash algorithm: sha1 [ 3.464233] ima: No architecture policies found [ 3.465272] evm: Initialising EVM extended attributes: [ 3.466405] evm: security.selinux [ 3.467224] evm: security.ima [ 3.467733] evm: security.capability [ 3.468602] evm: HMAC attrs: 0x1 [ 3.470323] rtc_cmos 00:05: setting system clock to 2026-08-08 11:56:43 UTC (1786190203) [ 3.476670] debug: unmapping init [mem 0xffffffffb9003000-0xffffffffb91fffff] [ 3.479800] debug: unmapping init [mem 0xffffffffb7d82000-0xffffffffb8058fff] [ 3.492118] Write protecting the kernel read-only data: 28672k [ 3.495036] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 3.496835] debug: unmapping init [mem 0xffffffffb6d14000-0xffffffffb6dfffff] [ 3.524239] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.530320] systemd[1]: Detected virtualization kvm. [ 3.531970] systemd[1]: Detected architecture x86-64. [ 3.533224] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.558970] systemd[1]: No hostname configured. [ 3.560050] systemd[1]: Set hostname to . [ 3.561333] random: systemd: uninitialized urandom read (16 bytes read) [ 3.562707] systemd[1]: Initializing machine ID from random generator. [ 3.716733] random: systemd: uninitialized urandom read (16 bytes read) [ 3.718926] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.722294] random: systemd: uninitialized urandom read (16 bytes read) [ 3.724265] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.727411] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.309491] device-mapper: uevent: version 1.0.3 [ 4.311647] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [ 4.910153] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.933833] virtio_net virtio0 ens2: renamed from eth0 [ 5.050513] scsi host0: ata_piix [ 5.065471] scsi host1: ata_piix [ 5.067305] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.069965] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.553219] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.980650] random: crng init done [ 9.981517] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.244603] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.356850] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.583599] SELinux: Disabled at runtime. [ 11.646810] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.655155] systemd[1]: Detected virtualization kvm. [ 11.656974] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.140938] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.144605] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.148581] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.153256] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.155903] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.161421] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.165451] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.247231] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.549511] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.877609] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.885241] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.065129] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.075397] EDAC sbridge: Ver: 1.1.2 [ 14.423747] Key type dns_resolver registered [ 14.749884] NFS: Registering the id_resolver key type [ 14.752089] Key type id_resolver registered [ 14.753660] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ ***] A start job is running for Configur…-only root support (8s / no limit) [ **] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg116-server login: [ 67.597627] libcfs: loading out-of-tree module taints kernel. [ 67.672047] Key type ._llcrypt registered [ 67.675805] Key type .llcrypt registered [ 67.806248] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 88.538031] hrtimer: interrupt took 9017599 ns [ 91.048483] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 93.212845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.235303] alg: No test for adler32 (adler32-zlib) [ 95.113289] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 96.261428] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 98.263326] Key type lgssc registered [ 100.148837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.104896] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 169.314749] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 186.095404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 186.143281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.616507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.677942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.845754] Lustre: lustre-MDT0000: new disk, initializing [ 188.054417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.092392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.313080] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.054025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.326136] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 211.371798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 211.380568] Lustre: Skipped 1 previous similar message [ 211.451818] Lustre: lustre-MDT0001: new disk, initializing [ 211.550744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.601264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.615968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.467804] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.898087] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.648695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.820962] Lustre: lustre-OST0000: new disk, initializing [ 234.823755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.828169] Lustre: 8451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.946406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.139978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.152846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.213268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 242.895500] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.504101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.746478] Lustre: lustre-OST0001: new disk, initializing [ 261.757509] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 261.773408] Lustre: 9538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.929486] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.845107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.852956] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.907520] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.395393] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 286.468316] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 293.146518] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 300.872351] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.368109] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing yml_node [ 312.028255] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 315.264070] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 318.424467] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 320.267710] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 08:01:59 EDT 2026 [ 341.112864] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 343.161814] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 360.931494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.942917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.955033] Lustre: Skipped 3 previous similar messages [ 360.986026] Lustre: Skipped 3 previous similar messages [ 366.048705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.071699] Lustre: Skipped 3 previous similar messages [ 371.168859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 371.176357] Lustre: Skipped 3 previous similar messages [ 374.242759] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 374.477684] Lustre: server umount lustre-MDT0000 complete [ 380.132212] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190580 with bad export cookie 11853218050086845470 [ 380.144320] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.155449] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 381.408228] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.413475] LustreError: 9292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.414300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 381.419049] Lustre: Skipped 1 previous similar message [ 381.440189] LustreError: 9292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 385.441980] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 385.443333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 385.474444] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 385.509912] Lustre: Skipped 2 previous similar messages [ 386.873653] Lustre: server umount lustre-MDT0001 complete [ 391.708832] Lustre: server umount lustre-OST0000 complete [ 396.724577] Lustre: server umount lustre-OST0001 complete [ 404.492216] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 414.891552] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 458.358476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.627301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 458.665929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 458.749826] Lustre: lustre-MDT0000: new disk, initializing [ 458.814698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.832814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 464.209509] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 476.323420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.541416] Lustre: 15095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 476.611946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 476.615707] Lustre: Skipped 1 previous similar message [ 476.731316] Lustre: lustre-MDT0001: new disk, initializing [ 476.969133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 477.057414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 477.080789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 483.038639] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 494.727745] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 503.769502] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.876271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 514.336585] Lustre: lustre-OST0000: new disk, initializing [ 514.344693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 514.364759] Lustre: 17064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 514.537685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.350633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 516.379520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 516.615849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 522.576784] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 535.600074] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 542.656766] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 542.962054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 548.498741] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 548.743299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 552.425844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.427842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.436626] LustreError: Skipped 1 previous similar message [ 552.444408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 552.498119] Lustre: Skipped 1 previous similar message [ 557.437571] Lustre: server umount lustre-OST0000 complete [ 562.657406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.385319] Lustre: server umount lustre-MDT0000 complete [ 572.911674] LustreError: 15107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.939807] LustreError: 15107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 574.000030] LustreError: 15891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190774 with bad export cookie 11853218050086847003 [ 574.019311] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.033363] LustreError: 15891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 574.510208] Lustre: server umount lustre-MDT0001 complete [ 581.618604] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:06:19 (1786190779) [ 584.352763] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 586.510799] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:06:25 (1786190785) [ 594.936388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.719203] LustreError: 19475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.906879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.064156] LustreError: 19476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.736319] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 610.283729] LustreError: 19476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.300181] LustreError: 19476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 613.506658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.223940] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 628.246947] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 634.989858] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 643.279619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 651.087015] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 659.841801] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 675.818930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.819341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 675.836636] Lustre: Skipped 1 previous similar message [ 675.838549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 675.838570] Lustre: Skipped 5 previous similar messages [ 680.546624] Lustre: server umount lustre-OST0000 complete [ 685.547182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 685.557417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.576333] Lustre: Skipped 1 previous similar message [ 690.992396] Lustre: server umount lustre-MDT0000 complete [ 691.173931] LustreError: 19500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 691.190169] LustreError: 19500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 694.971077] LustreError: 19456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190895 with bad export cookie 11853218050086847885 [ 695.001378] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 695.446676] Lustre: server umount lustre-MDT0001 complete [ 709.617604] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 712.628961] Key type lgssc unregistered [ 712.983236] LNet: 23035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.990503] LNetError: 23035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 714.023054] LNet: Removed LNI 192.168.201.116@tcp [ 715.379269] Key type .llcrypt unregistered [ 715.380957] Key type ._llcrypt unregistered [ 741.367810] Key type ._llcrypt registered [ 741.370468] Key type .llcrypt registered [ 741.542668] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 742.816596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 742.822775] alg: No test for adler32 (adler32-zlib) [ 744.025366] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 744.242088] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 745.903524] Key type lgssc registered [ 747.277414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.989509] loop: module loaded [ 758.088033] loop0: detected capacity change from 0 to 204800000 [ 758.416258] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 758.437136] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 758.997072] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 766.787364] loop0: detected capacity change from 0 to 204800000 [ 767.012518] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 767.024224] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 767.048477] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 773.305338] loop0: detected capacity change from 0 to 204800000 [ 780.229783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 780.269812] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.651312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 781.738283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 781.922253] Lustre: lustre-MDT0000: new disk, initializing [ 782.077461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.154921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 787.884443] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 798.193049] loop1: detected capacity change from 0 to 204800000 [ 804.588759] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.942981] Lustre: lustre-OST07c6: new disk, initializing [ 804.946231] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 804.950184] Lustre: Skipped 1 previous similar message [ 804.954054] Lustre: 26225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 805.026050] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 811.763979] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 812.121989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 812.129689] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 812.172683] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 817.915498] Lustre: 25212:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 817.928697] Lustre: 25212:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 817.939489] Lustre: 25212:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 817.951315] Lustre: 25212:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 817.967878] Lustre: 25212:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 817.978247] Lustre: 25212:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 839.793483] Lustre: 26889:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 839.800150] Lustre: 26889:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 839.805251] Lustre: 26889:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 839.809742] Lustre: 26889:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 839.814983] Lustre: 26889:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 839.819635] Lustre: 26889:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 845.100135] Lustre: 25213:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 845.114671] Lustre: 25213:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 845.127467] Lustre: 25213:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 845.146980] Lustre: 25213:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.161039] Lustre: 25213:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 845.176183] Lustre: 25213:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.192290] Lustre: 25213:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 845.209309] Lustre: 25213:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.216472] Lustre: 25213:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 845.222824] Lustre: 25213:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.229877] Lustre: 25213:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 845.235525] Lustre: 25213:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.480311] Lustre: 25211:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 848.491987] Lustre: 25211:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 848.506436] Lustre: 25211:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 848.521187] Lustre: 25211:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.526397] Lustre: 25211:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 848.531081] Lustre: 25211:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.542090] Lustre: 25211:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 848.547997] Lustre: 25211:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.556702] Lustre: 25211:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 848.563835] Lustre: 25211:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 848.568428] Lustre: 25211:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 848.579613] Lustre: 25211:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.401886] Lustre: 25211:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 853.409952] Lustre: 25211:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 853.416530] Lustre: 25211:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 853.422332] Lustre: 25211:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.429150] Lustre: 25211:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 853.435116] Lustre: 25211:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.445723] Lustre: 25211:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 853.459077] Lustre: 25211:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.467935] Lustre: 25211:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 853.482615] Lustre: 25211:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.502599] Lustre: 25211:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 853.509476] Lustre: 25211:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 861.808676] Lustre: 26889:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 861.822922] Lustre: 26889:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 861.833872] Lustre: 26889:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 861.841837] Lustre: 26889:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 861.853280] Lustre: 26889:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 861.858769] Lustre: 26889:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 861.864921] Lustre: 26889:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 861.870804] Lustre: 26889:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 861.876644] Lustre: 26889:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 861.881745] Lustre: 26889:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 861.894963] Lustre: 26889:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 861.904743] Lustre: 26889:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 914.400532] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 914.413827] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.426951] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 919.408848] Lustre: server umount lustre-OST07c6 complete [ 930.513689] Lustre: server umount lustre-MDT0000 complete [ 947.378549] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 950.251956] Key type lgssc unregistered [ 950.472626] LNet: 28785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.479644] LNetError: 28785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.530945] LNet: Removed LNI 192.168.201.116@tcp [ 952.512345] Key type .llcrypt unregistered [ 952.517526] Key type ._llcrypt unregistered [ 970.182688] Key type ._llcrypt registered [ 970.184591] Key type .llcrypt registered [ 970.285361] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 971.004375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 971.060337] alg: No test for adler32 (adler32-zlib) [ 972.112668] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 972.391981] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 974.087232] Key type lgssc registered [ 975.009345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 982.686528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 982.708268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.068486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.782529] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 994.283396] LustreError: 30057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.308207] LustreError: 30057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 996.926472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.242350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1002.113699] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1010.242338] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1015.589786] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1023.493388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1024.011619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.176351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1029.659327] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1036.548727] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1054.690721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1054.706427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.723690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1057.766706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1057.775461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1057.796744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.670500] Lustre: server umount lustre-OST0000 complete [ 1068.001896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.005832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.008955] Lustre: Skipped 1 previous similar message [ 1068.018737] Lustre: Skipped 2 previous similar messages [ 1070.834600] Lustre: server umount lustre-MDT0000 complete [ 1073.127307] LustreError: 30058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.143274] LustreError: 30058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1074.355758] LustreError: 30038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191274 with bad export cookie 1483547259137178406 [ 1074.360082] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1074.361631] LustreError: 30038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1074.781480] Lustre: server umount lustre-MDT0001 complete [ 1085.216704] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1088.127709] Key type lgssc unregistered [ 1088.371056] LNet: 33797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.382995] LNetError: 33797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.397982] LNet: Removed LNI 192.168.201.116@tcp [ 1089.188818] Key type .llcrypt unregistered [ 1089.193221] Key type ._llcrypt unregistered [ 1109.571785] Key type ._llcrypt registered [ 1109.573640] Key type .llcrypt registered [ 1109.682230] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1110.595764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.872539] alg: No test for adler32 (adler32-zlib) [ 1111.914398] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1112.164219] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1113.879160] Key type lgssc registered [ 1114.978420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1122.208364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1122.220329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.610059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.349162] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1133.026097] LustreError: 35076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.051175] LustreError: 35076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1137.490017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.983696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1142.267771] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1149.615930] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1155.156955] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1161.462792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1161.778655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.554331] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1169.017674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1174.216652] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1184.737859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1184.740321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1184.751135] Lustre: Skipped 1 previous similar message [ 1184.757958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1189.858502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1189.865512] Lustre: Skipped 2 previous similar messages [ 1194.979697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1194.985681] Lustre: Skipped 1 previous similar message [ 1199.071306] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1199.253963] Lustre: server umount lustre-OST0000 complete [ 1204.706715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1204.720517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.740792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.985600] Lustre: server umount lustre-MDT0000 complete [ 1210.338225] LustreError: 35805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.351785] LustreError: 35805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1212.455059] LustreError: 35057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191412 with bad export cookie 13850603818606708082 [ 1212.457396] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1212.464499] LustreError: 35057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1212.695545] Lustre: server umount lustre-MDT0001 complete [ 1223.496811] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1226.296315] Key type lgssc unregistered [ 1226.553937] LNet: 38614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.561371] LNetError: 38614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.578351] LNet: Removed LNI 192.168.201.116@tcp [ 1227.341656] Key type .llcrypt unregistered [ 1227.344290] Key type ._llcrypt unregistered [ 1250.283886] Key type ._llcrypt registered [ 1250.285763] Key type .llcrypt registered [ 1250.408370] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1251.678303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1251.848499] alg: No test for adler32 (adler32-zlib) [ 1252.978576] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1253.319839] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1255.015231] Key type lgssc registered [ 1255.877170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1262.853335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1262.870166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.448319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.584501] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1274.854915] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.866723] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1276.005304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.300202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.377542] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1286.484324] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1291.098743] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1297.311500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1297.654214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.638141] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1308.640694] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1309.944357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1319.691726] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1320.318621] Lustre: server umount lustre-MDT0000 complete [ 1320.418303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.420319] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1320.437873] Lustre: Skipped 2 previous similar messages [ 1320.444135] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1323.662828] LustreError: 40023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191523 with bad export cookie 5661464622814289791 [ 1323.674654] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.919353] Lustre: server umount lustre-MDT0001 complete [ 1335.917647] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.16@tcp arrived at 1786191535 with bad export cookie 5661464622814290456 [ 1354.581393] Lustre: server umount lustre-OST0000 complete [ 1368.649074] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1370.820266] Key type lgssc unregistered [ 1371.075192] LNet: 43907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.078820] LNetError: 43907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.088355] LNet: Removed LNI 192.168.201.116@tcp [ 1371.688973] Key type .llcrypt unregistered [ 1371.691438] Key type ._llcrypt unregistered [ 1394.323769] Key type ._llcrypt registered [ 1394.325690] Key type .llcrypt registered [ 1394.415272] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1395.461829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1395.612129] alg: No test for adler32 (adler32-zlib) [ 1396.662557] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1396.866782] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1398.511450] Key type lgssc registered [ 1399.361128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1405.727877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1405.744069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.116040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.856581] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1417.563793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.697333] LustreError: 45328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.710847] LustreError: 45328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1417.903085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1421.696531] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1427.511535] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1431.616714] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1436.710811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1436.926318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.815898] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1446.156771] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1452.343039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1461.993984] Lustre: lustre-OST0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 1462.761772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.767671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1462.774910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.774278] Lustre: server umount lustre-OST0000 complete [ 1484.769477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1484.777268] LustreError: Skipped 1 previous similar message [ 1484.790436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1484.804641] Lustre: Skipped 1 previous similar message [ 1484.808482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1484.816201] Lustre: Skipped 1 previous similar message [ 1487.329110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.332354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.236178] Lustre: server umount lustre-MDT0000 complete [ 1492.448864] LustreError: 45328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.468342] LustreError: 45328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1492.486102] LustreError: 45310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191692 with bad export cookie 3410869894198705753 [ 1492.494694] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.497825] LustreError: 45310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1498.826629] Lustre: server umount lustre-MDT0001 complete [ 1507.341047] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1509.641551] Key type lgssc unregistered [ 1509.943543] LNet: 49061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.949927] LNetError: 49061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.966906] LNet: Removed LNI 192.168.201.116@tcp [ 1510.719808] Key type .llcrypt unregistered [ 1510.722666] Key type ._llcrypt unregistered [ 1530.945495] Key type ._llcrypt registered [ 1530.947452] Key type .llcrypt registered [ 1531.022922] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1532.198531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1532.224645] alg: No test for adler32 (adler32-zlib) [ 1533.138851] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1533.273834] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1534.911225] Key type lgssc registered [ 1535.961925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1542.463379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1542.478243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.761544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.913143] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1553.896677] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.915781] LustreError: 50474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1554.680281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.942228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1558.393366] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1564.102742] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1568.572515] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1573.816290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.070561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.035724] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1584.446958] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1588.396935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1593.831437] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1596.133210] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1608.005548] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-08 8h23m26s [ 1609.278877] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1611.234382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1611.246204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.257901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.320617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.327891] Lustre: Skipped 5 previous similar messages [ 1617.238918] Lustre: server umount lustre-MDT0000 complete [ 1617.624282] LustreError: 50471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.236604] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1623.769563] LustreError: 50470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1623.779219] LustreError: 50470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1625.727358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.837479] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1626.028725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.087247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1627.056814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1629.381449] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1632.238162] Lustre: DEBUG MARKER: Wait for df (36687) ... [ 1633.760700] Lustre: DEBUG MARKER: done [ 1637.349282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.365994] Lustre: Skipped 2 previous similar messages [ 1637.372726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1637.379153] Lustre: Skipped 1 previous similar message [ 1642.465568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.476579] Lustre: Skipped 2 previous similar messages [ 1643.367610] Lustre: server umount lustre-OST0000 complete [ 1647.586886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1647.587895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.596769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.614936] Lustre: Skipped 2 previous similar messages [ 1652.627275] Lustre: server umount lustre-MDT0000 complete [ 1653.729399] LustreError: 50471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1653.757524] LustreError: 50471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1655.856173] LustreError: 50456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191855 with bad export cookie 16402152219804210253 [ 1655.857817] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1655.866949] LustreError: 50456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1656.124603] Lustre: server umount lustre-MDT0001 complete [ 1665.518155] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1667.640909] Key type lgssc unregistered [ 1667.875760] LNet: 55397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.880409] LNetError: 55397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.902702] LNet: Removed LNI 192.168.201.116@tcp [ 1669.550485] Key type .llcrypt unregistered [ 1669.552617] Key type ._llcrypt unregistered [ 1695.048848] Key type ._llcrypt registered [ 1695.050961] Key type .llcrypt registered [ 1695.144466] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1695.976926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1696.014955] alg: No test for adler32 (adler32-zlib) [ 1696.979627] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1697.129966] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1698.743260] Key type lgssc registered [ 1699.492298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.658851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1704.676471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.865483] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1705.888599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1705.985678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.911209] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1713.896849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.946472] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1713.957102] Lustre: 57581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1714.133501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1716.641303] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1721.145802] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1724.826958] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1728.924933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1729.085343] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1729.217286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.797774] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1735.268975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1737.092987] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1742.940446] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-08 8h25m42s [ 1743.964373] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1749.049623] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1750.769721] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1750.820726] LustreError: 57590:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1771.257637] Lustre: lustre-MDT0000: Client 88cfed9e-bccd-40be-8bf9-9285b38751b5 (at 192.168.201.16@tcp) reconnecting [ 1790.895297] LustreError: 57590:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1792.210229] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1793.678642] Lustre: DEBUG MARKER: done [ 1796.576956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1796.582408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.587984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1796.596264] Lustre: Skipped 1 previous similar message [ 1796.607804] Lustre: Skipped 1 previous similar message [ 1801.701169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1801.705844] Lustre: Skipped 1 previous similar message [ 1802.444560] Lustre: server umount lustre-OST0000 complete [ 1806.306743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1806.317353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.329089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.106761] Lustre: server umount lustre-MDT0000 complete [ 1813.649548] LustreError: 57576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192013 with bad export cookie 678311255851314589 [ 1813.652367] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.657450] LustreError: 57576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1813.853667] Lustre: server umount lustre-MDT0001 complete [ 1821.029297] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 1823.122531] Key type lgssc unregistered [ 1823.288665] LNet: 61915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.291900] LNetError: 61915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.305243] LNet: Removed LNI 192.168.201.116@tcp [ 1823.929234] Key type .llcrypt unregistered [ 1823.931494] Key type ._llcrypt unregistered [ 1851.842545] Key type ._llcrypt registered [ 1851.844291] Key type .llcrypt registered [ 1851.920049] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 1852.497902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.610366] alg: No test for adler32 (adler32-zlib) [ 1853.573382] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1853.730970] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 1855.399967] Key type lgssc registered [ 1856.148807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1858.334181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1858.345556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.547367] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1859.556245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1859.634889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.155673] Lustre: Failing over lustre-MDT0000 [ 1860.307241] Lustre: server umount lustre-MDT0000 complete [ 1864.798978] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:27:44 (1786192064) [ 1869.832876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.233810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1873.122362] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1879.280044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.325103] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1879.338615] Lustre: 64696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1879.504135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1882.259035] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1886.661398] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1889.714439] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1893.662340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1893.767617] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1893.898673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.740741] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1899.062854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1901.724440] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1906.673984] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1914.340067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.345417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.346757] Lustre: Skipped 1 previous similar message [ 1914.359804] Lustre: Skipped 2 previous similar messages [ 1916.735700] Lustre: server umount lustre-MDT0000 complete [ 1919.288649] LustreError: 64690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192119 with bad export cookie 4402650102462469976 [ 1919.292093] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.297776] LustreError: 64690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1919.458492] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.458767] LustreError: 64704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.466666] Lustre: Skipped 1 previous similar message [ 1919.477317] LustreError: 64704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1919.539897] Lustre: server umount lustre-MDT0001 complete [ 1921.704839] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1924.681937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1929.878895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.033331] LustreError: 63620:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000099bbbd35 x1872958118202240/t0(0) o250->MGC192.168.201.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1944.365404] LustreError: 66545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.454802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.539325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1947.689586] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1949.670952] LustreError: 66546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.767095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.108366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1956.952418] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 1959.217446] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1961.888718] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1965.679931] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1969.453874] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1974.754889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.756155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1974.762293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1975.952636] Lustre: server umount lustre-MDT0000 complete [ 1978.275638] LustreError: 65433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192178 with bad export cookie 4402650102462472258 [ 1978.277365] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.286764] LustreError: 65433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1978.458684] Lustre: server umount lustre-MDT0001 complete [ 1982.621721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1988.449108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.167442] Lustre: 63623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786192179/real 1786192179] req@00000000df4505c3 x1872958118223744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786192195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1995.186832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.198266] Lustre: Skipped 2 previous similar messages [ 1999.071138] Lustre: 63622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786192183/real 1786192183] req@000000001d8c85eb x1872958118223872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786192199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.290916] Lustre: 63621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786192184/real 1786192184] req@00000000d8032132 x1872958118224000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786192200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2003.597283] LustreError: 68487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.651822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.688634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2004.389016] Lustre: 63623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786192188/real 1786192188] req@0000000027905e96 x1872958118224256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786192204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2006.107481] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2010.084347] Lustre: 63624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786192193/real 1786192193] req@00000000314c17ac x1872958118224384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786192209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2011.108705] LustreError: 71145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.121571] LustreError: 71145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2011.156129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.903425] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2014.505774] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2018.482371] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2021.981404] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2026.992852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2026.997355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.007880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2027.010257] Lustre: Skipped 2 previous similar messages [ 2030.055397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.868198] Lustre: server umount lustre-MDT0000 complete [ 2033.985604] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192234 with bad export cookie 4402650102462473637 [ 2033.997268] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.998496] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2034.190737] Lustre: server umount lustre-MDT0001 complete [ 2038.141372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2040.081287] Lustre: DEBUG MARKER: files compared the same [ 2051.623154] Lustre: server umount lustre-OST0000 complete [ 2061.330083] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2063.092862] Key type lgssc unregistered [ 2063.288255] LNet: 74423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.293413] LNetError: 74423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.304249] LNet: Removed LNI 192.168.201.116@tcp [ 2063.893274] Key type .llcrypt unregistered [ 2063.895063] Key type ._llcrypt unregistered [ 2078.652803] Key type ._llcrypt registered [ 2078.654053] Key type .llcrypt registered [ 2078.713498] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2079.314895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2079.419456] alg: No test for adler32 (adler32-zlib) [ 2080.351663] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2080.556222] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2082.175515] Key type lgssc registered [ 2082.742947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2086.877324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2086.888276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.123556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.471812] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2094.966248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.143991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2097.429689] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2101.353324] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2104.388733] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2108.051625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2108.186870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.242975] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug +malloc all [ 2113.317884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2115.683448] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2125.794099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.794104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2125.794351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2125.805392] Lustre: Skipped 1 previous similar message [ 2127.273669] Lustre: server umount lustre-OST0000 complete [ 2130.915215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2130.918120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.918681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.923012] Lustre: Skipped 1 previous similar message [ 2130.929267] Lustre: Skipped 2 previous similar messages [ 2135.576960] Lustre: server umount lustre-MDT0000 complete [ 2136.031603] LustreError: 75817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.039392] LustreError: 75817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2137.658119] LustreError: 75797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192337 with bad export cookie 18404190296711014367 [ 2137.660367] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.664916] LustreError: 75797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2137.947917] Lustre: server umount lustre-MDT0001 complete [ 2143.246756] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2144.792875] Key type lgssc unregistered [ 2144.964522] LNet: 79348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.967705] LNetError: 79348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.975740] LNet: Removed LNI 192.168.201.116@tcp [ 2145.407736] Key type .llcrypt unregistered [ 2145.411101] Key type ._llcrypt unregistered [ 2159.547709] Key type ._llcrypt registered [ 2159.549610] Key type .llcrypt registered [ 2159.619262] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2160.119731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.151994] alg: No test for adler32 (adler32-zlib) [ 2161.070203] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2161.211131] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2162.831206] Key type lgssc registered [ 2163.487710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2167.755837] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2167.766938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2183.199332] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2188.319326] LustreError: 80715:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.116@tcp: failed processing log, type 4: rc = -110 [ 2214.943186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.324195] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2222.121122] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2226.413958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.475659] Lustre: *** cfs_fail_loc=706, val=0*** [ 2226.694700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.748588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2229.111554] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2233.878815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.076016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2236.075447] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2239.780166] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2242.927233] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2244.580251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2244.580996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.584114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.587653] Lustre: Skipped 1 previous similar message [ 2249.698757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2249.701493] Lustre: Skipped 2 previous similar messages [ 2254.824169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2254.826879] Lustre: Skipped 1 previous similar message [ 2258.399792] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2258.471839] Lustre: server umount lustre-OST0000 complete [ 2265.056848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.057032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2265.057589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.062553] Lustre: Skipped 1 previous similar message [ 2266.668673] Lustre: server umount lustre-MDT0000 complete [ 2268.700141] LustreError: 83372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192468 with bad export cookie 3520172496542973767 [ 2268.706241] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.711531] LustreError: 83372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2268.919520] Lustre: server umount lustre-MDT0001 complete [ 2275.006489] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2276.547927] Key type lgssc unregistered [ 2276.715720] LNet: 84329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.721852] LNetError: 84329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.729429] LNet: Removed LNI 192.168.201.116@tcp [ 2277.127224] Key type .llcrypt unregistered [ 2277.129323] Key type ._llcrypt unregistered [ 2291.249903] Key type ._llcrypt registered [ 2291.251668] Key type .llcrypt registered [ 2291.302492] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2291.798226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2291.855100] alg: No test for adler32 (adler32-zlib) [ 2292.818186] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2292.997516] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2294.639221] Key type lgssc registered [ 2295.332877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2299.657427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2299.677042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.257426] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2307.595460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.931469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2310.368771] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2313.187553] LustreError: 86409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.200748] LustreError: 86409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2315.202892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2315.354309] LustreError: 86408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.363506] LustreError: 86408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2315.378130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.411073] LustreError: 86413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.428462] LustreError: 86413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2318.696942] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2322.915624] LustreError: 86412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.927957] LustreError: 86412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2323.291734] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.299429] Lustre: MGS: Client 2a40dc09-5114-4e88-b6ab-10fc3df307f1 (at 0@lo) reconnecting [ 2323.314843] LustreError: 86423:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2323.441369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.485427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2325.659624] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2338.787796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2338.788729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.793877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2338.813550] Lustre: Skipped 1 previous similar message [ 2341.804773] Lustre: server umount lustre-OST0000 complete [ 2343.905099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2343.905157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.916959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.919160] Lustre: Skipped 2 previous similar messages [ 2349.025560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.029629] Lustre: Skipped 1 previous similar message [ 2349.646053] Lustre: server umount lustre-MDT0000 complete [ 2351.537110] Lustre: server umount lustre-MDT0001 complete [ 2353.238906] Lustre: server umount MGS complete [ 2358.877361] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:35:58 (1786192558) [ 2373.570545] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 2378.329465] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2396.501530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.569790] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2404.118043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.263217] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2404.290572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2404.330275] Lustre: lustre-MDT0001: new disk, initializing [ 2404.369644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2404.387266] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2404.391948] Lustre: 93570:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2404.396916] LustreError: 93570:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2406.372221] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2412.129152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2412.251434] Lustre: lustre-OST0000: new disk, initializing [ 2412.254964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2412.259100] Lustre: 94335:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2412.299352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.732154] Lustre: 94373:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001f0c81cf x1872958579576704/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786192629 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2415.546438] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2420.508932] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2423.803900] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2423.810676] Lustre: MGS: Client 67f19095-e5a2-418d-87ab-93af58bab8e1 (at 0@lo) reconnecting [ 2423.828185] Lustre: 92840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2423.837172] LustreError: 93556:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2423.841930] LustreError: 93556:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2423.846572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2423.894660] Lustre: lustre-MDT0000: new disk, initializing [ 2423.942607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.961771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2424.011858] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2424.993860] Lustre: 95405:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002dba7a20 x1872958579582976/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786192641 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2425.985186] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2434.020265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2434.024894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2434.151789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2458.598208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.602789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.609790] Lustre: Skipped 2 previous similar messages [ 2463.713490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.719543] Lustre: Skipped 2 previous similar messages [ 2470.367584] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2470.460091] Lustre: server umount lustre-OST0000 complete [ 2473.954052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2473.956155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.958884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.963038] Lustre: Skipped 1 previous similar message [ 2473.970011] Lustre: Skipped 3 previous similar messages [ 2478.430762] Lustre: server umount lustre-MDT0000 complete [ 2479.078195] LustreError: 95828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.088695] LustreError: 95828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2486.594039] Lustre: server umount lustre-MDT0001 complete [ 2488.158517] Lustre: server umount MGS complete [ 2492.565092] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:38:12 (1786192692) [ 2500.153523] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2501.412857] Key type lgssc unregistered [ 2501.545471] LNet: 97904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.548945] LNetError: 97904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.561474] LNet: Removed LNI 192.168.201.116@tcp [ 2502.055324] Key type .llcrypt unregistered [ 2502.057736] Key type ._llcrypt unregistered [ 2510.940620] Key type ._llcrypt registered [ 2510.944424] Key type .llcrypt registered [ 2510.997525] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2511.523269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.596823] alg: No test for adler32 (adler32-zlib) [ 2512.489165] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2512.616205] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2514.215158] Key type lgssc registered [ 2514.763692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2518.632794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2518.642415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.901121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.857921] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2526.492804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.685779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2528.726384] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2532.101223] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2535.101258] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2538.363271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2538.589744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.611393] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2544.746326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2545.463548] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2555.363525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2555.367471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.373183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2557.410513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2557.410716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2557.419576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.677956] Lustre: server umount lustre-OST0000 complete [ 2562.530755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2562.532263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.533927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.533933] Lustre: Skipped 1 previous similar message [ 2562.545639] Lustre: Skipped 1 previous similar message [ 2567.648528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.653593] Lustre: Skipped 2 previous similar messages [ 2572.768283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.772593] Lustre: Skipped 1 previous similar message [ 2576.351250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2576.496634] Lustre: server umount lustre-MDT0000 complete [ 2577.887953] LustreError: 99103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.894071] LustreError: 99103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2578.342759] LustreError: 99837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192778 with bad export cookie 4831421527374018109 [ 2578.346994] LustreError: 99837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2578.347061] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2578.525656] Lustre: server umount lustre-MDT0001 complete [ 2583.940885] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2585.504915] Key type lgssc unregistered [ 2585.679500] LNet: 102636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.684901] LNetError: 102636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2586.726464] LNet: Removed LNI 192.168.201.116@tcp [ 2587.188625] Key type .llcrypt unregistered [ 2587.191152] Key type ._llcrypt unregistered [ 2596.061143] Key type ._llcrypt registered [ 2596.062918] Key type .llcrypt registered [ 2596.110908] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2596.599327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2596.667441] alg: No test for adler32 (adler32-zlib) [ 2597.560164] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2597.672195] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2599.273204] Key type lgssc registered [ 2599.884874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2603.158382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2603.160361] LustreError: 103698:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2603.166870] LustreError: 103699:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2603.169456] LustreError: 103698:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2603.172737] LustreError: 103698:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2603.176758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.178577] LustreError: 103698:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2604.450885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.213700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.359465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2610.540654] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2613.323292] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing lsmod [ 2614.884132] LustreError: 104900:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2614.905352] LustreError: 104904:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2614.908036] LustreError: 104900:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 2614.912734] LustreError: 104900:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2614.917784] LustreError: 104900:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2614.921283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2615.132207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.599611] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.609068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2618.609940] Lustre: Skipped 1 previous similar message [ 2618.617674] Lustre: Skipped 2 previous similar messages [ 2622.694660] Lustre: server umount lustre-MDT0000 complete [ 2623.972912] LustreError: 104744:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.980584] LustreError: 104744:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2624.239677] LustreError: 103716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192824 with bad export cookie 3032183669480222548 [ 2624.244320] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.246053] LustreError: 103716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2624.759438] LustreError: 104167:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cfca13e2 x1872958898324992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2624.821532] Lustre: server umount lustre-MDT0001 complete [ 2636.359240] Lustre: server umount lustre-OST0000 complete [ 2640.336951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.582966] LustreError: 105970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.612343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.682315] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2645.984430] LustreError: 105971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.974590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.165313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.338940] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2653.307864] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2656.179132] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2659.783881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.940720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.822075] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2665.067183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2666.233528] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2675.680753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2675.684357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.689644] Lustre: Skipped 1 previous similar message [ 2675.692173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2677.730021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2677.731379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2677.736178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.883993] Lustre: server umount lustre-OST0000 complete [ 2683.363910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2683.367243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.376111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2683.378743] Lustre: Skipped 1 previous similar message [ 2687.970414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.971869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.198941] Lustre: server umount lustre-MDT0000 complete [ 2691.443314] LustreError: 105952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192891 with bad export cookie 3032183669480223234 [ 2691.443573] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2691.449152] LustreError: 105952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2691.621327] Lustre: server umount lustre-MDT0001 complete [ 2697.166892] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2698.765134] Key type lgssc unregistered [ 2698.996083] LNet: 109504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2699.004205] LNetError: 109504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.069289] LNet: Removed LNI 192.168.201.116@tcp [ 2700.571516] Key type .llcrypt unregistered [ 2700.574198] Key type ._llcrypt unregistered [ 2712.896225] Key type ._llcrypt registered [ 2712.897987] Key type .llcrypt registered [ 2712.949638] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2713.347873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2713.363085] alg: No test for adler32 (adler32-zlib) [ 2714.248657] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2714.361841] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2715.967171] Key type lgssc registered [ 2716.474823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2719.791696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2719.800565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.985211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.728184] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2726.315770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.494806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2728.179284] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2731.392141] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2733.553115] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2736.437146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.602986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.078785] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2742.340416] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2746.858447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2750.446311] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2751.349899] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2752.992234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2752.995667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.000903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.090179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2757.092730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.093186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.669209] Lustre: server umount lustre-OST0000 complete [ 2762.208607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.208631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.209391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.209395] Lustre: Skipped 1 previous similar message [ 2762.213423] Lustre: Skipped 1 previous similar message [ 2766.192996] Lustre: server umount lustre-MDT0000 complete [ 2767.328679] LustreError: 110898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.336392] LustreError: 110898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2767.802357] LustreError: 110879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192967 with bad export cookie 11131414921575202890 [ 2767.806084] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.809097] LustreError: 110879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2767.971759] Lustre: server umount lustre-MDT0001 complete [ 2772.213462] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2773.433457] Key type lgssc unregistered [ 2773.569336] LNet: 114545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2773.572164] LNetError: 114545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2773.581363] LNet: Removed LNI 192.168.201.116@tcp [ 2773.942912] Key type .llcrypt unregistered [ 2773.944148] Key type ._llcrypt unregistered [ 2781.815280] Key type ._llcrypt registered [ 2781.816585] Key type .llcrypt registered [ 2781.863615] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2782.455870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.469125] alg: No test for adler32 (adler32-zlib) [ 2783.343499] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2783.454242] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2785.063162] Key type lgssc registered [ 2785.542170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2788.763419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2788.771909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.965139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.538036] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2795.176045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.378793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2797.096778] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2800.012017] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2802.212358] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2805.157017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.329698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.939612] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2810.929091] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2814.567415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2820.068845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.070493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2820.074113] Lustre: Skipped 1 previous similar message [ 2820.078445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2821.094505] Lustre: server umount lustre-OST0000 complete [ 2824.807848] Lustre: server umount lustre-MDT0000 complete [ 2825.184670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.185188] LustreError: 115745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.188872] Lustre: Skipped 1 previous similar message [ 2825.194014] LustreError: 115745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2826.422312] LustreError: 115726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193026 with bad export cookie 6373871338287291259 [ 2826.424907] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2826.426315] LustreError: 115726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2826.552845] Lustre: server umount lustre-MDT0001 complete [ 2830.541848] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2831.718659] Key type lgssc unregistered [ 2831.875607] LNet: 119272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2831.879654] LNetError: 119272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2831.889320] LNet: Removed LNI 192.168.201.116@tcp [ 2832.254238] Key type .llcrypt unregistered [ 2832.256592] Key type ._llcrypt unregistered [ 2842.763137] Key type ._llcrypt registered [ 2842.764362] Key type .llcrypt registered [ 2842.807361] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2843.282497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2843.291886] alg: No test for adler32 (adler32-zlib) [ 2844.237554] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2844.353890] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2845.952079] Key type lgssc registered [ 2846.527400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.337511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2850.350512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.581202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.419484] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2857.373645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.567945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2859.440208] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2862.865519] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2865.441110] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2868.124573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.276115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.496945] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2873.393723] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2876.463193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2887.561830] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2889.495455] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2889.499145] Lustre: Skipped 2 previous similar messages [ 2891.737945] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2891.741310] Lustre: Skipped 2 previous similar messages [ 2897.937439] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2897.953291] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2901.643051] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.16@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2901.648676] Lustre: Skipped 2 previous similar messages [ 2903.945594] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.16@tcp 192.168.0.[2,10]@tcp [ 2906.352101] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.16/24@tcp in log lustre-MDT0000 [ 2906.355319] Lustre: Skipped 2 previous similar messages [ 2912.261065] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.16/24@tcp [ 2912.263315] Lustre: Skipped 1 previous similar message [ 2917.344177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2917.347709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.353261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2917.410669] Lustre: server umount lustre-OST0000 complete [ 2919.009735] Lustre: server umount lustre-MDT0000 complete [ 2920.789550] LustreError: 120649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193120 with bad export cookie 15563607668188873958 [ 2920.793805] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2920.794239] LustreError: 120649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2930.201796] Lustre: server umount lustre-MDT0001 complete [ 2934.114294] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2935.431351] Key type lgssc unregistered [ 2935.581429] LNet: 125812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2935.585174] LNetError: 125812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2935.594951] LNet: Removed LNI 192.168.201.116@tcp [ 2935.963477] Key type .llcrypt unregistered [ 2935.964832] Key type ._llcrypt unregistered [ 2949.030510] Key type ._llcrypt registered [ 2949.031956] Key type .llcrypt registered [ 2949.095512] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 2949.631951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2949.648323] alg: No test for adler32 (adler32-zlib) [ 2950.524316] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2950.620567] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 2952.208125] Key type lgssc registered [ 2952.662932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.191736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2956.199172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.384411] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2957.389702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.395439] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.16/24@tcp [ 2958.888297] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2962.435644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.590228] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2962.600922] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.16/24@tcp [ 2962.621374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2964.256389] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2967.410046] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2969.649253] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2972.653370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.816444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.112104] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 2977.937579] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2982.055919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2987.490478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2987.492587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.494662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2987.498074] Lustre: Skipped 1 previous similar message [ 2988.699442] Lustre: server umount lustre-OST0000 complete [ 2992.412351] Lustre: server umount lustre-MDT0000 complete [ 2992.608678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.609277] LustreError: 127352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.613172] Lustre: Skipped 1 previous similar message [ 2992.618502] LustreError: 127352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2993.817483] LustreError: 127337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193193 with bad export cookie 8780045864136068647 [ 2993.819825] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2993.825085] LustreError: 127337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2993.941918] Lustre: server umount lustre-MDT0001 complete [ 2998.074746] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 2999.186695] Key type lgssc unregistered [ 2999.324240] LNet: 130945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2999.326672] LNetError: 130945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3000.358340] LNet: Removed LNI 192.168.201.116@tcp [ 3000.686146] Key type .llcrypt unregistered [ 3000.687172] Key type ._llcrypt unregistered [ 3010.638892] Key type ._llcrypt registered [ 3010.640038] Key type .llcrypt registered [ 3010.694038] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3017.613805] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3018.000518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3018.043044] alg: No test for adler32 (adler32-zlib) [ 3018.920519] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3019.027719] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3020.615123] Key type lgssc registered [ 3021.133587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3035.175240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3035.181384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.269804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3036.282071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3036.321966] Lustre: lustre-MDT0000: new disk, initializing [ 3036.354180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.363863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3037.928438] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3042.687586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.739048] Lustre: 134321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3042.765286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3042.767956] Lustre: Skipped 1 previous similar message [ 3042.805986] Lustre: lustre-MDT0001: new disk, initializing [ 3042.849498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3042.867045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3042.871414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3044.408557] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3048.902975] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3051.181781] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3054.120062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3054.243658] Lustre: lustre-OST0000: new disk, initializing [ 3054.245860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3054.250092] Lustre: 136285:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3054.291891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.497075] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3060.747968] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3062.368948] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3063.803390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3063.809839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3063.846928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3064.500174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3066.065749] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3066.144256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3068.897632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3068.898255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.907491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3068.910148] Lustre: Skipped 1 previous similar message [ 3072.985455] Lustre: server umount lustre-OST0000 complete [ 3078.623996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3078.626608] LustreError: Skipped 1 previous similar message [ 3078.628540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.633028] Lustre: Skipped 1 previous similar message [ 3078.635255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.666906] Lustre: server umount lustre-MDT0000 complete [ 3082.080666] LustreError: 134312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193282 with bad export cookie 17535531388360019858 [ 3082.084292] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3082.085768] LustreError: 134312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3082.243425] Lustre: server umount lustre-MDT0001 complete [ 3085.441315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.658202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.139243] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3090.375947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.510500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3092.087918] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3094.692812] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3096.629905] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3099.042919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3099.155909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.112183] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3103.714046] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3105.209374] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3112.440859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3113.936464] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3114.012540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3118.584585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.656592] Lustre: lustre-OST0001: new disk, initializing [ 3118.658547] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3118.662109] Lustre: 141932:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3118.698674] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3120.976268] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3124.455832] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3126.790298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3126.794212] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3126.829252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3127.595773] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3129.050636] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3129.131682] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3131.161339] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9181c307d800.ost_server_uuid 50 [ 3131.698960] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9181c307d800.ost_server_uuid in FULL state after 0 sec [ 3136.995237] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3136.995251] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.997497] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3136.997502] Lustre: Skipped 2 previous similar messages [ 3136.998547] LustreError: Skipped 1 previous similar message [ 3137.003224] Lustre: Skipped 2 previous similar messages [ 3139.919437] Lustre: server umount lustre-OST0001 complete [ 3142.114331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.114349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.115310] LustreError: 140335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.115324] LustreError: 140335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3142.116206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3142.116211] Lustre: Skipped 1 previous similar message [ 3142.134876] Lustre: Skipped 1 previous similar message [ 3147.232348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3147.234409] LustreError: 140332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.234892] Lustre: Skipped 2 previous similar messages [ 3147.239626] LustreError: 140332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3147.597697] Lustre: server umount lustre-OST0000 complete [ 3151.839953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3151.843110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.103262] Lustre: server umount lustre-MDT0000 complete [ 3156.508298] LustreError: 138495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193356 with bad export cookie 17535531388360020726 [ 3156.510986] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.513492] LustreError: 138495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3156.643239] Lustre: server umount lustre-MDT0001 complete [ 3163.699661] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3164.827522] Key type lgssc unregistered [ 3164.960510] LNet: 144953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3164.963521] LNetError: 144953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3164.976343] LNet: Removed LNI 192.168.201.116@tcp [ 3165.325621] Key type .llcrypt unregistered [ 3165.327054] Key type ._llcrypt unregistered [ 3179.360407] Key type ._llcrypt registered [ 3179.361460] Key type .llcrypt registered [ 3179.402576] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3185.895606] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3186.378046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3186.388625] alg: No test for adler32 (adler32-zlib) [ 3187.268766] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3187.369240] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3188.959198] Key type lgssc registered [ 3189.354175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3202.058218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3202.063796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3203.146927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3203.158178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3203.195435] Lustre: lustre-MDT0000: new disk, initializing [ 3203.216838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.223361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3204.751700] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3209.005259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.051455] Lustre: 149078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3209.076511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3209.079431] Lustre: Skipped 1 previous similar message [ 3209.098339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3209.120954] Lustre: lustre-MDT0001: new disk, initializing [ 3209.154128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3209.166917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3209.171846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3210.592761] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3214.382185] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3216.256450] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3218.604367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3218.720964] Lustre: lustre-OST0000: new disk, initializing [ 3218.722746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3218.724922] Lustre: 151045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3218.756743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.681127] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3224.424554] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3225.593597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3225.597932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3225.641291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3227.727296] Lustre: Failing over lustre-OST0000 [ 3227.796965] Lustre: server umount lustre-OST0000 complete [ 3240.866962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.931111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.934675] Lustre: Skipped 1 previous similar message [ 3240.992310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.999943] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3241.019588] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3243.077505] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3244.447023] Lustre: Failing over lustre-MDT0000 [ 3244.539253] Lustre: server umount lustre-MDT0000 complete [ 3245.024019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3245.026775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.047385] Lustre: 147307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786193430/real 1786193430] req@00000000528e648e x1872959516993920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786193446 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3246.049169] LustreError: 149089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3246.049494] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.064628] LustreError: 149089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3247.071170] Lustre: 147308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786193430/real 1786193430] req@00000000dac9c120 x1872959516994816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786193446 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3251.169248] LustreError: 149885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.175366] LustreError: 149885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3252.191262] Lustre: 147309:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786193435/real 1786193435] req@000000001398373a x1872959516995968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786193451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3256.288107] LustreError: 149088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.295227] LustreError: 149088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3257.616045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.659972] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3257.664321] Lustre: 149081:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3257.739961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3257.757779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3257.779362] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3259.183102] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3261.655178] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3262.950059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3262.957471] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3262.981383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3268.067182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.067329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3268.068433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.070829] Lustre: Skipped 2 previous similar messages [ 3268.074051] LustreError: Skipped 1 previous similar message [ 3270.030484] Lustre: server umount lustre-OST0000 complete [ 3273.184948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.185054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3273.189160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3273.192408] Lustre: Skipped 2 previous similar messages [ 3277.529939] Lustre: server umount lustre-MDT0000 complete [ 3278.303976] LustreError: 149089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.309379] LustreError: 149089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3278.798241] LustreError: 149872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193478 with bad export cookie 9855706240501056954 [ 3278.800864] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3278.802061] LustreError: 149872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3278.913266] Lustre: server umount lustre-MDT0001 complete [ 3282.567501] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3283.583333] Key type lgssc unregistered [ 3283.707422] LNet: 154692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3283.709609] LNetError: 154692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3283.722309] LNet: Removed LNI 192.168.201.116@tcp [ 3284.029048] Key type .llcrypt unregistered [ 3284.030507] Key type ._llcrypt unregistered [ 3304.467946] Key type ._llcrypt registered [ 3304.468958] Key type .llcrypt registered [ 3304.506241] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3304.892123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3304.934739] alg: No test for adler32 (adler32-zlib) [ 3305.806133] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3305.898947] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3307.479141] Key type lgssc registered [ 3307.832027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3310.387371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3310.394416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.538455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3312.834943] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3316.684821] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3318.883287] LustreError: 156238:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3318.887117] LustreError: 156238:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3318.889979] LustreError: 156238:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 3319.013558] Lustre: server umount lustre-MDT0000 complete [ 3324.282044] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3327.690824] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3340.504830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.588337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3340.600291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3340.635655] Lustre: lustre-MDT0000: new disk, initializing [ 3340.654419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3340.660613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3342.052657] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3346.549344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.592113] Lustre: 159533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3346.615657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3346.617755] Lustre: Skipped 1 previous similar message [ 3346.654604] Lustre: lustre-MDT0001: new disk, initializing [ 3346.687351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3346.700553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3346.704299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3348.097241] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3351.869854] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3353.763363] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3356.170715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3356.270908] Lustre: lustre-OST0000: new disk, initializing [ 3356.273520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3356.277198] Lustre: 161504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3356.314456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.476872] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3362.442159] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3365.366696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3365.370195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3365.400138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3369.341457] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3369.922309] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:52:50 (1786193570) [ 3437.023473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.023616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3437.027266] Lustre: Skipped 2 previous similar messages [ 3437.030240] Lustre: Skipped 2 previous similar messages [ 3440.603979] Lustre: server umount lustre-MDT0000 complete [ 3441.822761] LustreError: 159526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193641 with bad export cookie 14938761594813152378 [ 3441.822761] LustreError: 159528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193641 with bad export cookie 14938761594813152378 [ 3441.822782] LustreError: 159526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3441.822913] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3441.826258] LustreError: 159528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3442.143441] LustreError: 160338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.143499] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.147943] LustreError: 160338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3442.156080] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3446.175462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3446.175462] LustreError: 159541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3447.975705] Lustre: server umount lustre-MDT0001 complete [ 3455.368129] Lustre: server umount lustre-OST0000 complete [ 3458.223890] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3461.150950] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3472.268013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.347259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3472.356615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3472.396353] Lustre: lustre-MDT0000: new disk, initializing [ 3472.416692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.422172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3473.669633] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3477.734538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3477.777047] Lustre: 166132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3477.824450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3477.824520] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3477.825949] Lustre: Skipped 1 previous similar message [ 3477.866471] Lustre: lustre-MDT0001: new disk, initializing [ 3477.913210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3477.917070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3479.246283] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3482.808046] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3484.579749] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3486.797745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3486.900101] Lustre: lustre-OST0000: new disk, initializing [ 3486.902315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3486.905195] Lustre: 168100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3486.934111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.937157] Lustre: Skipped 1 previous similar message [ 3488.728261] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3489.716751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3489.719851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3489.742956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3492.276112] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3493.655343] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3493.739373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3495.117071] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3495.183983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3500.001508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.001667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3500.003158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3500.006052] Lustre: Skipped 1 previous similar message [ 3500.009244] LustreError: Skipped 1 previous similar message [ 3510.239109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3510.241920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3510.244225] Lustre: Skipped 4 previous similar messages [ 3510.290964] Lustre: server umount lustre-OST0000 complete [ 3513.823793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3513.826553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.588183] Lustre: server umount lustre-MDT0000 complete [ 3518.735021] LustreError: 168098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193718 with bad export cookie 14938761594813311194 [ 3518.737891] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3518.738073] LustreError: 168098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3518.844316] Lustre: server umount lustre-MDT0001 complete [ 3521.306552] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:55:21 (1786193721) [ 3526.175720] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3529.671815] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3541.272490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.358510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3541.368071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3541.407261] Lustre: lustre-MDT0000: new disk, initializing [ 3541.429679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3541.437469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3542.742405] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3546.750332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.789330] Lustre: 172849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3546.878354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3548.255509] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3551.823808] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3553.532808] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3555.790532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3555.909286] Lustre: 174826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3557.765476] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3561.297872] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3561.464368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3561.493277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3571.682575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3571.683411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.685174] LustreError: Skipped 1 previous similar message [ 3571.685841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3571.688708] Lustre: Skipped 2 previous similar messages [ 3571.694940] Lustre: Skipped 4 previous similar messages [ 3574.675064] Lustre: server umount lustre-OST0000 complete [ 3582.169233] Lustre: server umount lustre-MDT0000 complete [ 3583.421474] LustreError: 172842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193783 with bad export cookie 14938761594813312076 [ 3583.425232] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3583.429457] LustreError: 172842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3583.549330] Lustre: server umount lustre-MDT0001 complete [ 3586.264822] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:56:26 (1786193786) [ 3591.089874] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3594.319038] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3606.224968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.323649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3606.334385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3606.337602] Lustre: Skipped 3 previous similar messages [ 3606.376639] Lustre: lustre-MDT0000: new disk, initializing [ 3606.378282] Lustre: Skipped 2 previous similar messages [ 3606.401095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.403086] Lustre: Skipped 2 previous similar messages [ 3606.408806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3606.411724] Lustre: Skipped 2 previous similar messages [ 3607.859317] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3612.151432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.199152] Lustre: 179139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3612.308200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3613.687654] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3617.500792] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3619.400943] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3621.622037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.738675] Lustre: 181117:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3623.537815] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3627.106319] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3628.570711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3638.752271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3638.754181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.755221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3638.758922] Lustre: Skipped 3 previous similar messages [ 3638.760503] Lustre: Skipped 4 previous similar messages [ 3640.073771] Lustre: server umount lustre-OST0000 complete [ 3643.359735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3643.361513] LustreError: Skipped 1 previous similar message [ 3647.317751] Lustre: server umount lustre-MDT0000 complete [ 3648.491177] LustreError: 179134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193848 with bad export cookie 14938761594813313147 [ 3648.494230] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3648.496917] LustreError: 179134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3648.610144] Lustre: server umount lustre-MDT0001 complete [ 3652.162870] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3653.105795] Key type lgssc unregistered [ 3653.224436] LNet: 182976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3653.228137] LNetError: 182976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3653.237395] LNet: Removed LNI 192.168.201.116@tcp [ 3653.550356] Key type .llcrypt unregistered [ 3653.551783] Key type ._llcrypt unregistered [ 3662.141344] Key type ._llcrypt registered [ 3662.142201] Key type .llcrypt registered [ 3662.173440] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3662.510043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3662.529090] alg: No test for adler32 (adler32-zlib) [ 3663.372978] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3663.453321] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3665.031137] Key type lgssc registered [ 3665.363600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3667.943094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3667.950775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.112849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.470656] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3673.328306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3673.454055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3674.720780] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3677.026096] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3678.806219] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.962325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3681.064576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.989188] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3685.458117] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3694.374357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3708.895996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3708.898958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.903864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3709.920611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3709.921845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.921958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3713.548280] Lustre: server umount lustre-OST0000 complete [ 3715.040305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.045400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3715.047188] Lustre: Skipped 2 previous similar messages [ 3720.160220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.162505] Lustre: Skipped 1 previous similar message [ 3725.280030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.282842] Lustre: Skipped 1 previous similar message [ 3728.863098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3728.923476] Lustre: server umount lustre-MDT0000 complete [ 3730.170496] LustreError: 187169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193930 with bad export cookie 7835171885685032388 [ 3730.173647] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.175203] LustreError: 187169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3730.290250] Lustre: server umount lustre-MDT0001 complete [ 3734.006463] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 3735.020399] Key type lgssc unregistered [ 3735.150318] LNet: 187923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3735.152602] LNetError: 187923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3735.163275] LNet: Removed LNI 192.168.201.116@tcp [ 3735.482930] Key type .llcrypt unregistered [ 3735.484087] Key type ._llcrypt unregistered [ 3745.120927] Key type ._llcrypt registered [ 3745.122558] Key type .llcrypt registered [ 3745.170917] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3745.553073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3745.588953] alg: No test for adler32 (adler32-zlib) [ 3746.442953] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3746.536558] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 3748.119152] Key type lgssc registered [ 3748.523032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3751.176354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3751.183047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.313851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.666406] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3756.494750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.614051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3757.933589] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3760.336854] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3762.237757] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3764.604707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3764.717579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.584827] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3768.930937] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3776.999183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3782.114895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3782.115553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3782.116547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3782.118870] Lustre: Skipped 1 previous similar message [ 3782.122774] LustreError: Skipped 1 previous similar message [ 3784.414422] Lustre: lustre-OST0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 3784.416475] Lustre: Skipped 1 previous similar message [ 3787.234230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.235840] Lustre: Skipped 1 previous similar message [ 3787.530026] Lustre: server umount lustre-OST0000 complete [ 3789.694105] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9181c409f800.ost_server_uuid 50 [ 3790.198697] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3800.195928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.196681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3800.199644] Lustre: Skipped 1 previous similar message [ 3805.664637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3805.668858] Lustre: Skipped 2 previous similar messages [ 3806.355653] Lustre: server umount lustre-MDT0000 complete [ 3807.588375] LustreError: 189343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194007 with bad export cookie 15456783378874582308 [ 3807.592275] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3807.592411] LustreError: 189343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3807.723417] Lustre: server umount lustre-MDT0001 complete [ 3810.440450] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:00:10 (1786194010) [ 3812.675947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3812.833916] LustreError: 193228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.840296] LustreError: 193228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3812.860358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.260817] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3817.148699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.279949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3818.559619] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3820.790206] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3822.533704] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3824.620995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3826.532376] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3828.834328] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3830.958685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3831.026800] Lustre: lustre-OST0001: new disk, initializing [ 3831.028632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3831.031654] Lustre: 196065:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3831.060729] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3831.063740] Lustre: Skipped 1 previous similar message [ 3832.830953] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3835.900172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3835.903930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3835.927511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3836.366688] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3838.176491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3838.178353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.259776] Lustre: server umount lustre-OST0000 complete [ 3843.552563] LustreError: 195048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3843.556556] LustreError: 195048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3845.147350] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3845.230632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3846.560764] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3846.631379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3846.872299] LustreError: 196737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.992214] LustreError: 195047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.998813] LustreError: 195047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3858.913173] LustreError: 195047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.914052] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.914501] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3858.914505] Lustre: Skipped 2 previous similar messages [ 3858.917302] LustreError: 195047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3858.920262] Lustre: Skipped 1 previous similar message [ 3862.794286] Lustre: server umount lustre-OST0001 complete [ 3864.032580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.035725] Lustre: Skipped 1 previous similar message [ 3878.367099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3878.427046] Lustre: server umount lustre-MDT0000 complete [ 3879.392049] LustreError: 193228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.395721] LustreError: 193228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3879.539248] LustreError: 193211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194079 with bad export cookie 15456783378874583365 [ 3879.541385] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3879.542905] LustreError: 193211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3879.652757] Lustre: server umount lustre-MDT0001 complete [ 3886.679180] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:01:26 (1786194086) [ 3888.658186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.742039] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3888.751618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3888.813442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3889.987929] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3892.758335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.787129] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3892.797497] Lustre: 199490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3894.064466] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3896.421349] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3898.359248] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3900.655034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3900.721362] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3902.763953] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3904.161089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3905.217280] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3907.576308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3907.603111] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3909.608152] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3911.930302] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3913.187894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.188942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.191893] Lustre: Skipped 1 previous similar message [ 3913.194889] Lustre: Skipped 8 previous similar messages [ 3913.230794] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3918.910677] Lustre: server umount lustre-OST0000 complete [ 3922.648538] LustreError: 201338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.666526] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.666930] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.669091] LustreError: Skipped 1 previous similar message [ 3935.570488] Lustre: server umount lustre-OST0001 complete [ 3943.006826] Lustre: server umount lustre-MDT0000 complete [ 3944.292210] LustreError: 200236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194144 with bad export cookie 15456783378874584527 [ 3944.295145] LustreError: 200236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3944.295253] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3944.419625] Lustre: server umount lustre-MDT0001 complete [ 3952.350220] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:02:32 (1786194152) [ 3957.519428] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 3960.788591] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 3972.456922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.534827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3972.543608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3972.580592] Lustre: lustre-MDT0000: new disk, initializing [ 3972.600900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.602961] Lustre: Skipped 3 previous similar messages [ 3972.607639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3973.998411] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3978.131669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.169084] Lustre: 207551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3978.188957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3978.191246] Lustre: Skipped 1 previous similar message [ 3978.225501] Lustre: lustre-MDT0001: new disk, initializing [ 3978.266088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3978.269080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3979.612843] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3983.326049] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3985.143032] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3987.391883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.490199] Lustre: lustre-OST0000: new disk, initializing [ 3987.491602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3987.494215] Lustre: 209516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3989.468401] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 3993.193784] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3994.563840] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3995.643583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3995.646938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3995.688076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3996.679186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3998.078980] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3998.159646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4000.738397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4000.739634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.744290] Lustre: Skipped 3 previous similar messages [ 4000.746223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.747979] Lustre: Skipped 8 previous similar messages [ 4000.914463] Lustre: server umount lustre-OST0000 complete [ 4003.808021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4003.810385] LustreError: Skipped 1 previous similar message [ 4008.348285] Lustre: server umount lustre-MDT0000 complete [ 4009.624478] LustreError: 207544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194209 with bad export cookie 15456783378874586263 [ 4009.627133] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.628057] LustreError: 207544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4009.758537] Lustre: server umount lustre-MDT0001 complete [ 4012.622517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.773195] LustreError: 211743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.781359] LustreError: 211743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4014.169600] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4017.027676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4018.565436] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4021.031340] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4022.884089] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4025.076245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4027.036245] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4029.382370] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4030.865242] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4039.119979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4040.440653] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4040.510753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.720336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4047.253724] Lustre: server umount lustre-OST0000 complete [ 4049.264929] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4050.658567] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4074.338882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4074.449892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.452273] Lustre: Skipped 5 previous similar messages [ 4076.382282] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4078.841568] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4079.417511] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4082.506913] Lustre: server umount lustre-OST0000 complete [ 4085.849875] Lustre: server umount lustre-MDT0000 complete [ 4087.071392] LustreError: 211723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194287 with bad export cookie 15456783378874587131 [ 4087.074666] LustreError: 211723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4087.075222] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4087.194553] Lustre: server umount lustre-MDT0001 complete [ 4090.003780] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:04:50 (1786194290) [ 4092.313729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.465296] LustreError: 217753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.828438] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4096.595023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.107890] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4100.534711] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4102.508940] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4104.785272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4106.909451] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4109.277549] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4110.674637] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4118.898932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4120.227288] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4120.301111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4122.480510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4122.544855] Lustre: lustre-OST0001: new disk, initializing [ 4122.547378] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4122.550919] Lustre: 221177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4124.479417] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4128.140721] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4129.514532] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4132.865095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4132.868561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4132.898818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4133.656181] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4134.977022] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4135.052330] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4137.953605] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4137.955475] LustreError: Skipped 1 previous similar message [ 4137.956710] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.959153] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4137.959661] Lustre: Skipped 6 previous similar messages [ 4137.961820] Lustre: Skipped 6 previous similar messages [ 4141.758338] Lustre: server umount lustre-OST0001 complete [ 4143.701046] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4143.784846] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4145.270917] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4145.357416] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4168.931187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4170.955820] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4173.393753] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4173.927145] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4174.311474] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4174.316811] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4174.316898] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4179.424552] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4179.427839] LustreError: Skipped 2 previous similar messages [ 4180.664879] Lustre: server umount lustre-OST0001 complete [ 4198.672095] Lustre: server umount lustre-OST0000 complete [ 4214.239112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4214.303245] Lustre: server umount lustre-MDT0000 complete [ 4215.541235] LustreError: 221176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194415 with bad export cookie 15456783378874588062 [ 4215.545267] LustreError: 221176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4215.545418] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.744333] Lustre: server umount lustre-MDT0001 complete [ 4229.621369] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:07:09 (1786194429) [ 4231.764929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.856279] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4231.865263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4231.935342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.937185] Lustre: Skipped 5 previous similar messages [ 4233.366976] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4236.367554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.404420] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4236.415513] Lustre: 226434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4237.971525] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4240.554807] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4242.564471] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4244.958622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4245.029787] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4247.129141] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4249.737197] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4258.608774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4258.635031] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4260.782410] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4263.314232] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4263.850861] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4265.382048] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4265.465278] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4266.951201] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4267.026847] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4268.896307] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9181c4431800.ost_server_uuid 50 [ 4269.434859] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9181c4431800.ost_server_uuid in FULL state after 0 sec [ 4269.772441] Lustre: Permanently deactivating lustre-OST0001 [ 4269.778126] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4270.585950] Lustre: Permanently reactivating lustre-OST0001 [ 4275.172653] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4275.172936] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.176841] LustreError: Skipped 3 previous similar messages [ 4275.182106] Lustre: Skipped 7 previous similar messages [ 4277.695270] Lustre: server umount lustre-OST0001 complete [ 4280.290272] LustreError: 228278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.294825] LustreError: 228278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 4285.070236] Lustre: server umount lustre-OST0000 complete [ 4292.454853] Lustre: server umount lustre-MDT0000 complete [ 4293.688436] LustreError: 226426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194493 with bad export cookie 15456783378874589336 [ 4293.691335] LustreError: 226426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4293.692198] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4293.810352] Lustre: server umount lustre-MDT0001 complete [ 4301.488278] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:08:21 (1786194501) [ 4304.041423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.134895] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4304.145831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4304.149032] Lustre: Skipped 5 previous similar messages [ 4305.678853] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4308.489011] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.523106] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4308.534207] Lustre: 233115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4309.990436] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4312.442362] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4314.217264] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4316.463460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4316.565107] Lustre: Permanently deactivating lustre-OST0000 [ 4316.569910] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4318.261471] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4318.644603] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4321.044705] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4323.370623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4325.166692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4325.454527] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4327.905922] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4328.447511] Lustre: Permanently reactivating lustre-OST0000 [ 4334.075303] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4334.101077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4340.708489] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4340.711036] LustreError: Skipped 2 previous similar messages [ 4344.508385] Lustre: server umount lustre-OST0001 complete [ 4360.159131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4360.210661] Lustre: server umount lustre-OST0000 complete [ 4367.710488] Lustre: server umount lustre-MDT0000 complete [ 4368.971682] LustreError: 233150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194569 with bad export cookie 15456783378874590967 [ 4368.976307] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4368.976796] LustreError: 233150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4369.117596] Lustre: server umount lustre-MDT0001 complete [ 4372.847444] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4373.992639] Key type lgssc unregistered [ 4374.126394] LNet: 237970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4374.130844] LNetError: 237970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4374.140375] LNet: Removed LNI 192.168.201.116@tcp [ 4374.484055] Key type .llcrypt unregistered [ 4374.485146] Key type ._llcrypt unregistered [ 4383.243422] Key type ._llcrypt registered [ 4383.244656] Key type .llcrypt registered [ 4383.292375] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4383.850639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4383.861671] alg: No test for adler32 (adler32-zlib) [ 4384.708654] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4384.793419] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4386.375168] Key type lgssc registered [ 4386.728720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4389.976353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4389.982961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.132765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.419364] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4395.198870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.249194] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4395.261994] Lustre: 239403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4395.267422] Lustre: Permanently deactivating lustre-MDT0001 [ 4395.273933] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4395.376526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4396.634067] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4398.986404] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4400.626557] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4400.841508] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4403.017653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4403.144887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.098876] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4405.216850] LustreError: 241635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4405.219788] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4405.225043] LustreError: 241635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4407.531312] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4409.838209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4409.911245] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4411.765195] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4414.134314] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4414.950254] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4417.002964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4417.005441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4419.704581] Lustre: Permanently reactivating lustre-MDT0001 [ 4419.709665] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4419.711456] Lustre: Skipped 2 previous similar messages [ 4421.405194] Lustre: lustre-MDT0001: Client 88651c4b-d817-4044-b66e-a620b6dd555f (at 192.168.201.16@tcp) reconnecting [ 4423.533742] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4424.049200] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4425.737335] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.740855] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 4425.743283] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4425.749256] LustreError: 243461:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 4425.749704] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4425.753332] LustreError: 243461:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 4425.759404] LustreError: 243461:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.116@tcp: cfg command failed: rc = -108 [ 4425.761977] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 4425.761977] [ 4425.764479] LustreError: 239405:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4425.841932] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4425.923666] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4427.200242] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4427.263910] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4428.057080] Lustre: Permanently deactivating lustre-MDT0001 [ 4428.062302] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4428.064033] Lustre: Skipped 2 previous similar messages [ 4431.362505] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4439.010986] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4439.013340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.017678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.546021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.546131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.550327] Lustre: Skipped 2 previous similar messages [ 4440.552076] Lustre: Skipped 2 previous similar messages [ 4443.036528] Lustre: server umount lustre-MDT0000 complete [ 4444.303422] LustreError: 239393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194644 with bad export cookie 2812486561692377742 [ 4444.306141] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4444.307203] LustreError: 239393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4444.433256] Lustre: server umount lustre-MDT0001 complete [ 4451.707755] Lustre: server umount lustre-OST0000 complete [ 4455.454283] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:10:55 (1786194655) [ 4458.811262] Lustre: server umount lustre-OST0001 complete [ 4460.828568] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 4464.064959] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4475.135748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.224735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4475.229444] Lustre: Skipped 2 previous similar messages [ 4475.237390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4475.278263] Lustre: lustre-MDT0000: new disk, initializing [ 4475.299737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.307553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4476.659399] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4480.724093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.765405] Lustre: 247703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4480.785658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4480.788207] Lustre: Skipped 1 previous similar message [ 4480.828945] Lustre: lustre-MDT0001: new disk, initializing [ 4480.870106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4480.873146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4482.313190] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4486.064647] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4487.869761] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4490.150956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4490.279222] Lustre: lustre-OST0000: new disk, initializing [ 4490.281310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4490.283938] Lustre: 249676:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4490.313778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.316115] Lustre: Skipped 1 previous similar message [ 4492.323715] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4493.878396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4493.882846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4493.910896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4496.010936] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4496.698486] LustreError: 247712:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4500.870414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4500.949630] Lustre: lustre-OST0001: new disk, initializing [ 4500.951501] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4500.953981] Lustre: 250791:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4502.917645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4502.922191] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4502.951569] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4503.001497] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4506.739156] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.791072] LustreError: 247712:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4506.801905] LustreError: 247711:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4507.264978] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4516.895101] LustreError: 247711:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4518.370467] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4518.372200] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.372598] LustreError: Skipped 1 previous similar message [ 4518.378031] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4518.380264] Lustre: Skipped 2 previous similar messages [ 4522.206300] Lustre: lustre-OST0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 4523.585674] Lustre: server umount lustre-OST0001 complete [ 4528.611101] LustreError: 249659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.615747] LustreError: 249659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4533.731992] LustreError: 250143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.739584] LustreError: 250143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4538.852214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.853698] LustreError: 249659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.854854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4538.854860] Lustre: Skipped 2 previous similar messages [ 4538.856654] Lustre: Skipped 2 previous similar messages [ 4538.861982] LustreError: 249659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4541.707642] Lustre: server umount lustre-OST0000 complete [ 4549.088807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4549.090700] Lustre: Skipped 4 previous similar messages [ 4549.216409] Lustre: server umount lustre-MDT0000 complete [ 4550.424458] LustreError: 247697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194750 with bad export cookie 2812486561692380556 [ 4550.426651] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4550.429972] LustreError: 247697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4556.593804] Lustre: server umount lustre-MDT0001 complete [ 4560.227426] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4561.254261] Key type lgssc unregistered [ 4561.384373] LNet: 252820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4561.388467] LNetError: 252820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4561.401295] LNet: Removed LNI 192.168.201.116@tcp [ 4561.717100] Key type .llcrypt unregistered [ 4561.718156] Key type ._llcrypt unregistered [ 4575.553355] Key type ._llcrypt registered [ 4575.554289] Key type .llcrypt registered [ 4575.590280] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4575.989155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4576.011821] alg: No test for adler32 (adler32-zlib) [ 4576.855536] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4576.940127] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4578.527133] Key type lgssc registered [ 4578.920473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4581.549156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4581.557618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.643793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4582.652348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4582.716217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.079553] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4586.875571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.906173] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4586.919937] Lustre: 255005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4587.017559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4588.405806] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4590.788517] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4592.657591] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4594.840816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.906122] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4594.962816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.879408] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4598.482365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4599.275723] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4639.713408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.714173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.716671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4639.724364] Lustre: Skipped 1 previous similar message [ 4639.729224] Lustre: Skipped 1 previous similar message [ 4644.698246] Lustre: server umount lustre-OST0000 complete [ 4646.148359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4650.168724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4650.282155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4652.121841] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4654.472907] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4659.492731] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4659.496511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4659.496582] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4659.502438] Lustre: Skipped 1 previous similar message [ 4690.400558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4690.402583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.404103] LustreError: Skipped 2 previous similar messages [ 4690.408318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4690.411903] Lustre: Skipped 1 previous similar message [ 4693.848722] Lustre: server umount lustre-OST0000 complete [ 4695.520407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.520862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4695.523911] Lustre: Skipped 2 previous similar messages [ 4700.640628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4700.642561] Lustre: Skipped 2 previous similar messages [ 4705.760133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4705.763043] Lustre: Skipped 1 previous similar message [ 4709.343117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4709.406948] Lustre: server umount lustre-MDT0000 complete [ 4710.643213] LustreError: 255040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194910 with bad export cookie 9079091285312216148 [ 4710.646120] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4710.648284] LustreError: 255040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4710.759785] Lustre: server umount lustre-MDT0001 complete [ 4714.610088] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4715.616497] Key type lgssc unregistered [ 4715.737358] LNet: 260425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4715.739632] LNetError: 260425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4715.750300] LNet: Removed LNI 192.168.201.116@tcp [ 4716.080941] Key type .llcrypt unregistered [ 4716.081976] Key type ._llcrypt unregistered [ 4725.226617] Key type ._llcrypt registered [ 4725.227792] Key type .llcrypt registered [ 4725.261706] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4725.627849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4725.665968] alg: No test for adler32 (adler32-zlib) [ 4726.518276] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4726.603972] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4728.183100] Key type lgssc registered [ 4728.537568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4731.196185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4731.202442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.346640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.713604] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4736.518602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.640022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4737.971254] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4740.491461] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4742.422167] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4744.731274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4744.860929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.864280] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4749.309849] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4758.185138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4767.712497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4767.715048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.719175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4768.738815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4768.739197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.739421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4773.272209] Lustre: server umount lustre-OST0000 complete [ 4777.951957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4777.955401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.960389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.962409] Lustre: Skipped 1 previous similar message [ 4780.627611] Lustre: server umount lustre-MDT0000 complete [ 4781.842314] LustreError: 261797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194981 with bad export cookie 5628179353142675806 [ 4781.846105] LustreError: 261797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4781.847212] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4781.983106] Lustre: server umount lustre-MDT0001 complete [ 4785.712117] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4786.744041] Key type lgssc unregistered [ 4786.866320] LNet: 266142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4786.868933] LNetError: 266142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4786.876425] LNet: Removed LNI 192.168.201.116@tcp [ 4787.189186] Key type .llcrypt unregistered [ 4787.190134] Key type ._llcrypt unregistered [ 4794.452696] Key type ._llcrypt registered [ 4794.453901] Key type .llcrypt registered [ 4794.493544] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4794.844140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4794.852109] alg: No test for adler32 (adler32-zlib) [ 4795.713273] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4795.800291] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4797.375123] Key type lgssc registered [ 4797.743933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4800.341218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4800.348850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.506226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4802.790118] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4805.936237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.059604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4807.339493] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4809.748777] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4811.629964] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4813.838409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4813.949306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4815.791534] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4818.168359] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4826.279883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4836.834806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4836.837260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.837555] LustreError: Skipped 1 previous similar message [ 4836.841419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.844636] Lustre: Skipped 1 previous similar message [ 4838.537951] Lustre: server umount lustre-OST0000 complete [ 4841.954913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4841.954931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.955927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4841.966508] Lustre: Skipped 2 previous similar messages [ 4847.072126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4847.074485] Lustre: Skipped 2 previous similar messages [ 4852.192071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4852.193741] Lustre: Skipped 1 previous similar message [ 4854.239056] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4854.299664] Lustre: server umount lustre-MDT0000 complete [ 4855.517224] LustreError: 267321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195055 with bad export cookie 11983543737418896268 [ 4855.520083] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4855.520489] LustreError: 267321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4855.629147] Lustre: server umount lustre-MDT0001 complete [ 4859.097838] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4860.150220] Key type lgssc unregistered [ 4860.282385] LNet: 271037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4860.285245] LNetError: 271037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4860.293284] LNet: Removed LNI 192.168.201.116@tcp [ 4860.600994] Key type .llcrypt unregistered [ 4860.602040] Key type ._llcrypt unregistered [ 4869.747290] Key type ._llcrypt registered [ 4869.748944] Key type .llcrypt registered [ 4869.790440] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4870.320890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4870.331159] alg: No test for adler32 (adler32-zlib) [ 4871.191598] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4871.287364] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4872.879176] Key type lgssc registered [ 4873.342558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4876.082560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4876.089791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.226493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4878.536318] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4881.315862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4881.448791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4882.769081] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4885.350466] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4887.113306] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4889.314517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4889.421605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.200325] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4893.579644] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4902.762672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4912.096445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4912.099581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.105020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4913.121844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4913.122676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4913.123088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.124781] Lustre: Skipped 1 previous similar message [ 4917.646857] Lustre: server umount lustre-OST0000 complete [ 4922.335763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4922.337704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.341248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4925.017313] Lustre: server umount lustre-MDT0000 complete [ 4926.393388] LustreError: 272417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195126 with bad export cookie 16863277992244015812 [ 4926.396615] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.397355] LustreError: 272417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4926.525275] Lustre: server umount lustre-MDT0001 complete [ 4930.231520] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4931.256940] Key type lgssc unregistered [ 4931.379390] LNet: 276759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4931.382874] LNetError: 276759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4931.393319] LNet: Removed LNI 192.168.201.116@tcp [ 4931.696971] Key type .llcrypt unregistered [ 4931.698041] Key type ._llcrypt unregistered [ 4938.917480] Key type ._llcrypt registered [ 4938.918436] Key type .llcrypt registered [ 4938.951818] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 4939.311048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4939.345458] alg: No test for adler32 (adler32-zlib) [ 4940.205107] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4940.299118] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 4941.879174] Key type lgssc registered [ 4942.234883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4944.770370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4944.776076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.938779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4947.253582] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4950.165317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.289520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4951.611598] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4953.980225] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4955.863147] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4958.147832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.267945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4960.241943] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 4962.650659] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4970.602312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4981.219837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4981.220281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4981.221160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4981.223212] Lustre: Skipped 1 previous similar message [ 4982.859500] Lustre: server umount lustre-OST0000 complete [ 4986.337156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4986.337198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.340599] LustreError: Skipped 1 previous similar message [ 4986.347051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.351280] Lustre: Skipped 2 previous similar messages [ 4990.238198] Lustre: server umount lustre-MDT0000 complete [ 4991.449478] LustreError: 278731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195191 with bad export cookie 17286271736578198757 [ 4991.452760] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4991.455782] LustreError: 278731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4991.577839] Lustre: server umount lustre-MDT0001 complete [ 4995.327142] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 4996.341404] Key type lgssc unregistered [ 4996.471373] LNet: 281690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4996.473670] LNetError: 281690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4996.484274] LNet: Removed LNI 192.168.201.116@tcp [ 4996.796135] Key type .llcrypt unregistered [ 4996.797181] Key type ._llcrypt unregistered [ 5000.563477] Key type ._llcrypt registered [ 5000.564956] Key type .llcrypt registered [ 5000.605397] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5006.162826] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5012.280252] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5012.625419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5012.664070] alg: No test for adler32 (adler32-zlib) [ 5013.524944] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 5013.624731] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5015.215147] Key type lgssc registered [ 5015.591148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5026.859511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5026.867068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.949230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5027.959906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5027.995904] Lustre: lustre-MDT0000: new disk, initializing [ 5028.020151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5028.025946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5029.339654] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5033.433763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.479778] Lustre: 285212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5033.506861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5033.509483] Lustre: Skipped 1 previous similar message [ 5033.554225] Lustre: lustre-MDT0001: new disk, initializing [ 5033.587321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5033.599564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5033.603650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5034.947381] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5038.773966] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5040.691627] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5043.031576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5043.139419] Lustre: lustre-OST0000: new disk, initializing [ 5043.141818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5043.145635] Lustre: 287178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5043.178630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.160663] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5049.000288] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5049.340486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5049.343695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5049.381839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5050.515139] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5050.592262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5052.023343] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5052.103388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5054.437204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5054.437615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.441692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5054.443764] Lustre: Skipped 1 previous similar message [ 5054.448984] Lustre: Skipped 1 previous similar message [ 5058.904633] Lustre: server umount lustre-OST0000 complete [ 5064.672935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5064.673020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.675018] LustreError: Skipped 1 previous similar message [ 5064.680220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5064.682963] Lustre: Skipped 1 previous similar message [ 5066.330433] Lustre: server umount lustre-MDT0000 complete [ 5067.586159] LustreError: 285207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195267 with bad export cookie 10302935182067076342 [ 5067.589954] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5067.590983] LustreError: 285207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5067.709152] Lustre: server umount lustre-MDT0001 complete [ 5070.593976] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:21:10 (1786195270) [ 5072.843467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.040839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5074.360864] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5077.228304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.380469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5078.790439] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5081.183222] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5083.018849] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5085.243559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5087.253763] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5089.662505] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5103.072417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5103.074702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.079402] Lustre: Skipped 1 previous similar message [ 5103.081359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5109.070282] Lustre: server umount lustre-OST0000 complete [ 5113.311839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5113.315455] LustreError: Skipped 1 previous similar message [ 5113.317609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.322947] Lustre: Skipped 1 previous similar message [ 5113.325373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5113.326971] Lustre: Skipped 2 previous similar messages [ 5116.499205] Lustre: server umount lustre-MDT0000 complete [ 5117.745793] LustreError: 289569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195317 with bad export cookie 10302935182067077224 [ 5117.749565] LustreError: 289569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5117.749575] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.869760] Lustre: server umount lustre-MDT0001 complete [ 5121.471667] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5122.454423] Key type lgssc unregistered [ 5122.573315] LNet: 293146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5122.575571] LNetError: 293146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5122.582259] LNet: Removed LNI 192.168.201.116@tcp [ 5122.884214] Key type .llcrypt unregistered [ 5122.885164] Key type ._llcrypt unregistered [ 5131.076208] Key type ._llcrypt registered [ 5131.078309] Key type .llcrypt registered [ 5131.114742] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5137.414181] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5137.758769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5137.804373] alg: No test for adler32 (adler32-zlib) [ 5138.664736] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 5138.754825] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5140.335118] Key type lgssc registered [ 5140.661062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5154.938244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5154.943738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.027313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5156.035656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5156.071099] Lustre: lustre-MDT0000: new disk, initializing [ 5156.094626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.101086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5157.403395] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5161.457146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.505856] Lustre: 296939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5161.535888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5161.538733] Lustre: Skipped 1 previous similar message [ 5161.581905] Lustre: lustre-MDT0001: new disk, initializing [ 5161.620554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5161.636679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5161.640891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5163.035457] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5166.763568] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5168.524591] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5170.747572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5170.853228] Lustre: lustre-OST2710: new disk, initializing [ 5170.854673] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5170.856870] Lustre: 298908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5170.881408] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5172.720647] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5174.329650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5174.333205] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5174.371559] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5176.376555] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5178.596532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5178.664101] Lustre: lustre-OST03e8: new disk, initializing [ 5178.665841] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5178.668340] Lustre: 299969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5178.701862] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5180.557121] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5182.039387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5182.042614] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5182.083279] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5182.950611] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5185.515575] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5185.577793] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5186.827105] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5186.892877] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5188.141963] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5188.201512] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5189.557812] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5189.622212] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5192.674433] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.675432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5192.676776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5192.677634] Lustre: Skipped 2 previous similar messages [ 5196.765184] Lustre: server umount lustre-MDT0000 complete [ 5197.947369] LustreError: 299968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195397 with bad export cookie 8459241228661959017 [ 5197.950640] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5197.953244] LustreError: 299968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5198.077822] Lustre: server umount lustre-MDT0001 complete [ 5209.089126] Lustre: server umount lustre-OST2710 complete [ 5220.498741] Lustre: server umount lustre-OST03e8 complete [ 5226.348863] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5229.655182] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5241.236579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.315086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5241.325739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5241.369193] Lustre: lustre-MDT0000: new disk, initializing [ 5241.393285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.399482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5242.772598] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5246.812869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5246.856476] Lustre: 305209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5246.953451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5248.190504] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5251.835869] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5253.640921] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5255.796488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5255.889508] Lustre: lustre-OST0000: new disk, initializing [ 5255.890949] Lustre: Skipped 1 previous similar message [ 5255.892630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5255.894764] Lustre: Skipped 2 previous similar messages [ 5255.896545] Lustre: 307177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5255.927113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.929131] Lustre: Skipped 1 previous similar message [ 5257.766888] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5261.379346] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5262.724699] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5262.836589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5262.838926] Lustre: Skipped 1 previous similar message [ 5262.840380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5262.867325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5263.812423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5265.080271] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5265.147070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5267.937272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5267.938738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.939954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.939959] Lustre: Skipped 3 previous similar messages [ 5267.941054] LustreError: Skipped 1 previous similar message [ 5267.945339] Lustre: Skipped 1 previous similar message [ 5271.887381] Lustre: server umount lustre-OST0000 complete [ 5277.663930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5277.666442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5277.671288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5277.672981] Lustre: Skipped 1 previous similar message [ 5279.350238] Lustre: server umount lustre-MDT0000 complete [ 5280.716879] LustreError: 305246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195480 with bad export cookie 8459241228661960690 [ 5280.722110] LustreError: 305246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5280.722774] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5280.872587] Lustre: server umount lustre-MDT0001 complete [ 5283.718266] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:24:43 (1786195483) [ 5284.220068] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5284.808706] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:24:44 (1786195484) [ 5285.881217] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5294.215858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5294.296611] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5294.302876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5294.356490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.694760] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5298.467847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5298.499599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5298.508937] Lustre: 311036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5300.020183] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5315.873867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5315.940491] Lustre: Denying initial registration attempt from nid 192.168.201.116@tcp, specified as failover [ 5315.943525] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5315.946149] LustreError: 312486:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5315.948409] LustreError: 312486:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5315.950283] LustreError: 312486:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5315.970911] Lustre: server umount lustre-OST0000 complete [ 5315.972055] LustreError: 312486:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5319.135896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5319.138710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.142220] Lustre: Skipped 1 previous similar message [ 5319.143967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5319.146090] Lustre: Skipped 2 previous similar messages [ 5324.115946] Lustre: server umount lustre-MDT0000 complete [ 5325.278243] LustreError: 311775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195525 with bad export cookie 8459241228661961558 [ 5325.281045] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5325.281432] LustreError: 311775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5325.386718] Lustre: server umount lustre-MDT0001 complete [ 5328.981994] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5329.958374] Key type lgssc unregistered [ 5330.092378] LNet: 313607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5330.094383] LNetError: 313607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5330.101303] LNet: Removed LNI 192.168.201.116@tcp [ 5330.396139] Key type .llcrypt unregistered [ 5330.397204] Key type ._llcrypt unregistered [ 5334.172731] Key type ._llcrypt registered [ 5334.174186] Key type .llcrypt registered [ 5334.212335] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5334.579373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5334.609546] alg: No test for adler32 (adler32-zlib) [ 5335.464952] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 5335.549694] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5337.127117] Key type lgssc registered [ 5337.467510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5360.973604] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5363.978619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5363.985102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.067748] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5365.073708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5365.131304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5366.476112] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5369.345896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.376913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5369.387865] Lustre: 316345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5369.495900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5370.725249] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5372.997032] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5374.689234] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5389.793447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5389.857121] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5389.919306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.666610] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5393.919446] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5394.623492] Lustre: server umount lustre-OST0000 complete [ 5396.465665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.467786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5396.468636] Lustre: Skipped 1 previous similar message [ 5401.568316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5401.569994] Lustre: Skipped 2 previous similar messages [ 5406.688824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5406.690961] Lustre: Skipped 1 previous similar message [ 5410.271095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5410.333610] Lustre: server umount lustre-MDT0000 complete [ 5411.465128] LustreError: 316382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195611 with bad export cookie 9867519113592629600 [ 5411.468300] LustreError: 316382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5411.468740] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5411.580336] Lustre: server umount lustre-MDT0001 complete [ 5415.110970] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5416.048146] Key type lgssc unregistered [ 5416.161330] LNet: 319911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5416.163238] LNetError: 319911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5416.171285] LNet: Removed LNI 192.168.201.116@tcp [ 5416.462911] Key type .llcrypt unregistered [ 5416.463971] Key type ._llcrypt unregistered [ 5425.239221] Key type ._llcrypt registered [ 5425.240138] Key type .llcrypt registered [ 5425.276175] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5425.727768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5425.736466] alg: No test for adler32 (adler32-zlib) [ 5426.585120] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 5426.677294] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5428.255121] Key type lgssc registered [ 5428.622728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5431.204771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5431.212573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5432.385611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5433.784991] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5436.536877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5436.663280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5437.962419] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5440.216654] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5441.931303] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5446.115950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5446.252995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5448.068950] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5450.307254] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5466.847335] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 5467.108041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5467.113505] Lustre: Skipped 1 previous similar message [ 5469.660614] Lustre: server umount lustre-MDT0000 complete [ 5470.879776] LustreError: 323144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195670 with bad export cookie 2804581729401698535 [ 5470.884735] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5471.959310] LustreError: 323798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5471.964293] LustreError: 323798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5471.967335] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 5471.969289] Lustre: Skipped 3 previous similar messages [ 5472.224976] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.229626] Lustre: Skipped 1 previous similar message [ 5475.231867] LustreError: 321297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.232147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5475.236413] LustreError: 321297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5475.238171] Lustre: Skipped 1 previous similar message [ 5477.037847] Lustre: server umount lustre-MDT0001 complete [ 5479.113513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5481.897246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.775805] LustreError: 320614:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005c7d27d7 x1872961864753152/t0(0) o250->MGC192.168.201.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5495.881936] LustreError: 323131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.903478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.929687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5497.204473] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5499.914263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.000686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5500.003577] Lustre: Skipped 1 previous similar message [ 5500.044410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5501.334048] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5503.630277] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5505.188834] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5508.214155] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5509.936578] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5512.440729] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5515.747171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5515.747419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.752929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5515.754441] Lustre: Skipped 1 previous similar message [ 5519.695454] Lustre: server umount lustre-OST0000 complete [ 5520.864312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5520.864787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.869521] Lustre: Skipped 2 previous similar messages [ 5525.984216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5525.985690] Lustre: Skipped 3 previous similar messages [ 5535.199195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5535.274076] Lustre: server umount lustre-MDT0000 complete [ 5536.455717] LustreError: 321285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195736 with bad export cookie 2804581729401705689 [ 5536.460168] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5536.461829] LustreError: 321285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5536.569921] Lustre: server umount lustre-MDT0001 complete [ 5540.037435] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5541.068311] Key type lgssc unregistered [ 5541.188476] LNet: 327586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5541.190575] LNetError: 327586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5541.199364] LNet: Removed LNI 192.168.201.116@tcp [ 5541.496918] Key type .llcrypt unregistered [ 5541.498057] Key type ._llcrypt unregistered [ 5550.192283] Key type ._llcrypt registered [ 5550.193669] Key type .llcrypt registered [ 5550.227446] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5550.547262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5550.622759] alg: No test for adler32 (adler32-zlib) [ 5551.468681] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 5551.550688] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5553.127095] Key type lgssc registered [ 5553.421370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5555.696359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5555.704357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5556.845298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.055390] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5560.626443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5560.747606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5562.027709] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5564.191329] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5565.879233] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5570.912859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5570.916603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5570.919206] Lustre: Skipped 1 previous similar message [ 5576.159981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.161727] Lustre: Skipped 1 previous similar message [ 5581.279095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5581.280544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.283359] Lustre: Skipped 1 previous similar message [ 5581.337129] Lustre: server umount lustre-MDT0000 complete [ 5582.432125] LustreError: 328960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195782 with bad export cookie 18079640611061855712 [ 5582.434882] LustreError: 328960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5582.435548] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.543274] Lustre: server umount lustre-MDT0001 complete [ 5586.985646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.073625] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5587.083269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5587.157945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.452554] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5591.231019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.260194] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5591.269498] Lustre: 331568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5591.355730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5592.605699] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5594.891592] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5596.633580] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5601.314016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.382817] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5601.385753] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5601.389766] LustreError: 333451:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5601.392911] LustreError: 333451:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5601.395734] LustreError: 333451:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5601.416145] Lustre: server umount lustre-OST0000 complete [ 5601.417449] LustreError: 333451:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5606.241945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5606.321601] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5606.392689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.179415] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5610.390658] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5612.852665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5612.884348] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5612.906469] Lustre: lustre-OST0001: new disk, initializing [ 5612.907873] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5612.909912] Lustre: 334978:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5614.742917] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5616.133578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5616.138481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5616.166108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5616.189093] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5616.942436] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5621.219956] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5621.221130] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.222083] LustreError: Skipped 1 previous similar message [ 5621.225100] Lustre: Skipped 1 previous similar message [ 5621.226744] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5621.230356] Lustre: Skipped 1 previous similar message [ 5623.931905] Lustre: server umount lustre-OST0001 complete [ 5626.337919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5626.338215] LustreError: 335565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.338842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.339955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5626.339963] LustreError: Skipped 1 previous similar message [ 5626.347019] LustreError: 335565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5631.238919] Lustre: server umount lustre-OST0000 complete [ 5632.479724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5632.481634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5632.484453] Lustre: Skipped 1 previous similar message [ 5636.576063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5636.576587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5636.577782] Lustre: Skipped 2 previous similar messages [ 5638.549048] Lustre: server umount lustre-MDT0000 complete [ 5639.683098] LustreError: 331560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195839 with bad export cookie 18079640611061856251 [ 5639.685853] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5639.687974] LustreError: 331560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5639.801222] Lustre: server umount lustre-MDT0001 complete [ 5643.076211] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5643.976549] Key type lgssc unregistered [ 5644.086311] LNet: 336939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5644.088337] LNetError: 336939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5644.096265] LNet: Removed LNI 192.168.201.116@tcp [ 5644.380956] Key type .llcrypt unregistered [ 5644.382252] Key type ._llcrypt unregistered [ 5672.988678] Key type ._llcrypt registered [ 5672.989571] Key type .llcrypt registered [ 5673.030166] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 5678.959899] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5679.486238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5679.496289] alg: No test for adler32 (adler32-zlib) [ 5680.353383] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 5680.445437] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5682.023144] Key type lgssc registered [ 5682.372584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5692.717653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5692.722521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.802159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5693.812254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5693.846969] Lustre: lustre-MDT0000: new disk, initializing [ 5693.866452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.871855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5695.109768] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5698.978328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.015887] Lustre: 341842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5699.037403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5699.039067] Lustre: Skipped 1 previous similar message [ 5699.075146] Lustre: lustre-MDT0001: new disk, initializing [ 5699.103865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5699.113558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5699.116320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5700.304307] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5703.697097] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5705.361827] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5707.414695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5707.511030] Lustre: lustre-OST0000: new disk, initializing [ 5707.512390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5707.514473] Lustre: 343807:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5707.542433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.814525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5707.817189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5707.859771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5709.334042] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5712.903536] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5714.250315] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5714.316740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5715.840715] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5715.949602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5717.989485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5717.990761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5717.993962] LustreError: Skipped 1 previous similar message [ 5718.002219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5718.007748] Lustre: Skipped 1 previous similar message [ 5723.106540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5723.108504] Lustre: Skipped 1 previous similar message [ 5728.226639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.228299] Lustre: Skipped 1 previous similar message [ 5731.295158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5731.359590] Lustre: server umount lustre-OST0000 complete [ 5733.345904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5733.346794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5733.350031] Lustre: Skipped 2 previous similar messages [ 5738.464348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5738.466235] Lustre: Skipped 2 previous similar messages [ 5738.774885] Lustre: server umount lustre-MDT0000 complete [ 5739.920312] LustreError: 341833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195939 with bad export cookie 13645268080055105554 [ 5739.924139] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5739.925807] LustreError: 341833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5740.036758] Lustre: server umount lustre-MDT0001 complete [ 5742.590185] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:32:22 (1786195942) [ 5744.915431] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:32:25 (1786195945) [ 5747.628043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.791611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5749.028499] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5751.679361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.801756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5753.097172] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5755.352361] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5757.099373] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5759.341372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.335377] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5763.609517] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5777.317633] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5777.860492] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5778.362613] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5778.857124] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5779.437126] Lustre: Failing over lustre-MDT0000 [ 5779.621264] Lustre: server umount lustre-MDT0000 complete [ 5782.495775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5782.498209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5782.502168] LustreError: 346558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.509843] LustreError: 346558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5783.008283] LustreError: 346577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.012713] LustreError: 346577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5784.284963] LustreError: 347998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.128084] LustreError: 346559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.132237] LustreError: 346559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5792.355810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.392476] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5792.475691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.477392] Lustre: Skipped 1 previous similar message [ 5792.494224] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5793.764680] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5794.519653] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5797.864095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5797.873746] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5797.891302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5799.663339] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5800.188596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5802.048598] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5802.704109] Lustre: Failing over lustre-MDT0000 [ 5802.949978] Lustre: server umount lustre-MDT0000 complete [ 5802.975859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5802.977403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.978982] LustreError: 346577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.981196] Lustre: Skipped 4 previous similar messages [ 5802.986805] LustreError: 346577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5813.216684] LustreError: 346553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.223745] LustreError: 346553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5815.588564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.623262] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5815.709956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.737731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5816.945082] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5820.119634] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5820.904307] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5820.906023] Lustre: Skipped 2 previous similar messages [ 5820.912679] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5820.930906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5822.582156] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5823.074299] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5826.016325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5826.017976] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.019474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5826.023695] Lustre: Skipped 2 previous similar messages [ 5826.025129] Lustre: Skipped 1 previous similar message [ 5831.380738] Lustre: server umount lustre-MDT0000 complete [ 5832.475081] LustreError: 348399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196032 with bad export cookie 13645268080055108235 [ 5832.477823] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5832.478053] LustreError: 348399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5832.584079] Lustre: server umount lustre-MDT0001 complete [ 5843.987165] Lustre: server umount lustre-OST0000 complete [ 5848.854637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.004946] LustreError: 352908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.012117] LustreError: 352908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5849.028873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.349635] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5853.201796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.570429] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5856.960978] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5858.776947] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5861.021040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5863.041845] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5865.496491] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5874.469425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5878.751746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5878.753646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.756987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5878.758486] Lustre: Skipped 4 previous similar messages [ 5883.210436] Lustre: server umount lustre-OST0000 complete [ 5889.503851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5889.506178] LustreError: Skipped 1 previous similar message [ 5890.707190] Lustre: server umount lustre-MDT0000 complete [ 5891.880169] LustreError: 352934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196091 with bad export cookie 13645268080055108956 [ 5891.883459] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5891.885282] LustreError: 352934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5892.001173] Lustre: server umount lustre-MDT0001 complete [ 5895.707848] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 5896.724366] Key type lgssc unregistered [ 5896.860364] LNet: 356462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5896.862875] LNetError: 356462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5896.872397] LNet: Removed LNI 192.168.201.116@tcp [ 5897.201706] Key type .llcrypt unregistered [ 5897.202744] Key type ._llcrypt unregistered [ 5906.979166] Key type ._llcrypt registered [ 5906.980053] Key type .llcrypt registered [ 5907.013415] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 5907.437223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5907.444132] alg: No test for adler32 (adler32-zlib) [ 5908.288017] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 5908.373452] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 5909.959158] Key type lgssc registered [ 5910.345837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5912.859041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5912.864420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.009321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.294067] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5918.151538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5918.268774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5919.574931] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5922.006364] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5923.799297] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5926.078945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5926.200180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.150893] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5930.527355] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5939.495323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5944.801513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.802100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5944.805232] Lustre: Skipped 1 previous similar message [ 5949.921301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5949.924039] Lustre: Skipped 4 previous similar messages [ 5950.943631] Lustre: server umount lustre-MDT0000 complete [ 5952.152435] LustreError: 357936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196152 with bad export cookie 10851495681858860352 [ 5952.155886] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5952.157521] LustreError: 357936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5952.275707] Lustre: server umount lustre-MDT0001 complete [ 5963.798157] Lustre: server umount lustre-OST0000 complete [ 5968.144453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.295166] LustreError: 361680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.301490] LustreError: 361680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5968.317881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.610394] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5972.336490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.467034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5973.779787] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5976.172847] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5977.871547] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5980.083348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5982.095802] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 5984.398549] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5993.509882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6003.167770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6003.169869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.172877] Lustre: Skipped 1 previous similar message [ 6003.174301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6007.574468] Lustre: server umount lustre-MDT0000 complete [ 6008.712151] LustreError: 361661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196208 with bad export cookie 10851495681858861451 [ 6008.715230] LustreError: 361661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6008.715956] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.819885] Lustre: server umount lustre-MDT0001 complete [ 6020.253344] Lustre: server umount lustre-OST0000 complete [ 6025.013848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.159360] LustreError: 365609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.180025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6025.181753] Lustre: Skipped 1 previous similar message [ 6026.488514] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6029.210116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.628527] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6033.044450] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6034.871649] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6037.040172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6038.960506] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6041.379390] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6050.470053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6054.880406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6054.882345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.885159] Lustre: Skipped 2 previous similar messages [ 6054.886731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6054.889378] Lustre: Skipped 2 previous similar messages [ 6055.908079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6059.342141] Lustre: server umount lustre-OST0000 complete [ 6061.024701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.025570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.027689] Lustre: Skipped 2 previous similar messages [ 6061.029235] Lustre: Skipped 2 previous similar messages [ 6066.646150] Lustre: server umount lustre-MDT0000 complete [ 6067.843423] LustreError: 365590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196267 with bad export cookie 10851495681858862459 [ 6067.848312] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6067.964552] Lustre: server umount lustre-MDT0001 complete [ 6071.667559] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6072.650261] Key type lgssc unregistered [ 6072.767389] LNet: 369164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6072.769393] LNetError: 369164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6072.781321] LNet: Removed LNI 192.168.201.116@tcp [ 6073.067746] Key type .llcrypt unregistered [ 6073.068749] Key type ._llcrypt unregistered [ 6082.594218] Key type ._llcrypt registered [ 6082.595221] Key type .llcrypt registered [ 6082.630241] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6083.057649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6083.071669] alg: No test for adler32 (adler32-zlib) [ 6083.917293] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6083.997614] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6085.575079] Key type lgssc registered [ 6085.927822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6088.503559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6088.508825] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.510994] LustreError: 370581:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6088.517888] LustreError: 370581:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6088.520082] LustreError: 370581:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6088.522248] LustreError: 370581:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6088.524450] LustreError: 370581:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6090.943780] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.945955] LustreError: 370979:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6090.953855] LustreError: 370979:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6090.956734] LustreError: 370979:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6090.959790] LustreError: 370979:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6090.963220] LustreError: 370979:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6097.399508] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6098.394917] Key type lgssc unregistered [ 6098.514302] LNet: 371984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6098.516644] LNetError: 371984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6098.527281] LNet: Removed LNI 192.168.201.116@tcp [ 6098.801333] Key type .llcrypt unregistered [ 6098.802438] Key type ._llcrypt unregistered [ 6104.457771] Key type ._llcrypt registered [ 6104.458690] Key type .llcrypt registered [ 6104.501669] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6110.835438] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6111.213815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6111.226207] alg: No test for adler32 (adler32-zlib) [ 6112.088982] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6112.184929] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6113.767144] Key type lgssc registered [ 6114.139851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6125.688158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6125.692489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.768837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6126.779310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6126.815313] Lustre: lustre-MDT0000: new disk, initializing [ 6126.835793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.842110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6128.017665] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6132.006238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.052878] Lustre: 375115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6132.080593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6132.082244] Lustre: Skipped 1 previous similar message [ 6132.119694] Lustre: lustre-MDT0001: new disk, initializing [ 6132.156979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6132.168831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6132.172460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6133.475791] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6137.025422] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6138.912452] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6141.148472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6141.258906] Lustre: lustre-OST0000: new disk, initializing [ 6141.260471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6141.263186] Lustre: 377078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6141.291175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.227453] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6146.951469] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6148.379090] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6150.654232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6150.657320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6150.698715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6151.528463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6152.977531] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6153.064995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6155.745795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6155.746312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6155.751642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6155.753265] Lustre: Skipped 1 previous similar message [ 6159.765702] Lustre: server umount lustre-OST0000 complete [ 6162.912204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6162.915408] LustreError: Skipped 1 previous similar message [ 6162.917562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.923873] Lustre: Skipped 1 previous similar message [ 6162.926735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6165.984248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6165.984832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.196985] Lustre: server umount lustre-MDT0000 complete [ 6168.452610] LustreError: 375106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196368 with bad export cookie 11607014939554872279 [ 6168.457446] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6168.458604] LustreError: 375106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6168.577242] Lustre: server umount lustre-MDT0001 complete [ 6171.302289] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6171.842464] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:39:32 (1786196372) [ 6174.017955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.200170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.500155] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6178.250969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.381880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6179.851526] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6182.227553] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6184.071526] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6186.297586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.406707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.225695] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6190.526381] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6192.784134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.858806] Lustre: lustre-OST0001: new disk, initializing [ 6192.861371] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6192.864968] Lustre: 382387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6194.768772] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6195.710960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6195.714161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6195.742559] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6198.447426] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6200.800077] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6200.800340] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.802339] LustreError: Skipped 1 previous similar message [ 6200.802818] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6200.805729] Lustre: Skipped 1 previous similar message [ 6200.807140] Lustre: Skipped 1 previous similar message [ 6202.364149] Lustre: server umount lustre-OST0001 complete [ 6203.021863] LustreError: 381768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.025938] LustreError: 381768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6204.899083] LustreError: 381373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.903829] LustreError: 381373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6210.018864] LustreError: 381768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.023641] LustreError: 381768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6215.138611] LustreError: 383310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.138970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.140038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6215.140154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6215.143321] LustreError: 383310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6215.146300] Lustre: Skipped 1 previous similar message [ 6220.258848] LustreError: 383310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.262986] LustreError: 383310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6220.493465] Lustre: server umount lustre-OST0000 complete [ 6224.351832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6224.353886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.357317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6224.358859] Lustre: Skipped 3 previous similar messages [ 6227.877974] Lustre: server umount lustre-MDT0000 complete [ 6229.321216] LustreError: 379538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196429 with bad export cookie 11607014939554873147 [ 6229.327718] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6229.329288] LustreError: 379538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6235.522685] Lustre: server umount lustre-MDT0001 complete [ 6239.124627] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6240.263911] Key type lgssc unregistered [ 6240.403715] LNet: 384375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6240.410165] LNetError: 384375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6240.422637] LNet: Removed LNI 192.168.201.116@tcp [ 6240.832299] Key type .llcrypt unregistered [ 6240.833894] Key type ._llcrypt unregistered [ 6261.436492] Key type ._llcrypt registered [ 6261.437531] Key type .llcrypt registered [ 6261.475331] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6261.797138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6261.866038] alg: No test for adler32 (adler32-zlib) [ 6262.720051] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6262.809110] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6264.391149] Key type lgssc registered [ 6264.877557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6267.989111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6267.996292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.091482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6269.097455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6269.155164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6270.867042] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6274.850957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.899624] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6274.915047] Lustre: 387146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6275.064183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6276.567728] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6279.168068] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6281.095498] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6283.745547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6283.746435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6283.751353] Lustre: Skipped 1 previous similar message [ 6283.756942] Lustre: Skipped 1 previous similar message [ 6288.864620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6288.868028] Lustre: Skipped 1 previous similar message [ 6293.984276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6293.986251] Lustre: Skipped 1 previous similar message [ 6296.031230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6296.119133] Lustre: server umount lustre-MDT0000 complete [ 6297.895255] LustreError: 387138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196497 with bad export cookie 3664515903257057403 [ 6297.900700] LustreError: 387138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6297.902701] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6298.050846] Lustre: server umount lustre-MDT0001 complete [ 6301.747663] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:41:41 (1786196501) [ 6304.946799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6305.236431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.041161] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6310.370646] LustreError: 389659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.380795] LustreError: 389659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6310.804032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.006460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6312.963646] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6316.035533] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6318.120385] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6320.952616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6321.079474] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6321.199171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6323.380368] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6326.529569] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6336.131344] Lustre: Permanently deactivating lustre-OST0000 [ 6336.138387] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6345.204429] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6346.720860] LustreError: 392790:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6346.726047] LustreError: 392790:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6346.729402] LustreError: 392790:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6347.358669] LustreError: 392838:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6347.361432] LustreError: 392838:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6354.127427] Lustre: server umount lustre-OST0000 complete [ 6355.124198] LustreError: 393088:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6355.129078] LustreError: 393088:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6356.960666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6356.964291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.968941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6362.592899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6362.595653] Lustre: Skipped 3 previous similar messages [ 6370.271094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6370.353541] Lustre: server umount lustre-MDT0000 complete [ 6371.561365] LustreError: 392350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196571 with bad export cookie 3664515903257057970 [ 6371.564189] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6371.564831] LustreError: 392350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6381.094286] Lustre: server umount lustre-MDT0001 complete [ 6384.454963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.746794] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6386.621715] LustreError: 394245:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6388.847666] Lustre: 394583:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.116@tcp [ 6389.530908] Lustre: server umount MGS complete [ 6393.247103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.404495] LustreError: 395306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.408616] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6393.410745] Lustre: Skipped 1 previous similar message [ 6393.428613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6394.634844] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6397.377852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.464056] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6398.650414] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6400.934173] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6402.681239] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6403.281365] Lustre: Permanently reactivating lustre-OST0000 [ 6403.287138] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6403.290279] Lustre: Skipped 2 previous similar messages [ 6431.199133] LustreError: 397183:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6433.898495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6434.055183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6434.058502] Lustre: Skipped 1 previous similar message [ 6436.689655] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6439.704858] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6455.776508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6455.781852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.787596] Lustre: Skipped 1 previous similar message [ 6455.789108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6455.791510] Lustre: Skipped 2 previous similar messages [ 6456.802685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6457.806211] Lustre: server umount lustre-OST0000 complete [ 6461.920577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.925563] Lustre: Skipped 2 previous similar messages [ 6465.184851] Lustre: server umount lustre-MDT0000 complete [ 6466.625382] LustreError: 395287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196666 with bad export cookie 3664515903257059006 [ 6466.629097] LustreError: 395287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6466.629418] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6466.760545] Lustre: server umount lustre-MDT0001 complete [ 6470.286597] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6471.427300] Key type lgssc unregistered [ 6471.564403] LNet: 400078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6471.568283] LNetError: 400078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6471.578527] LNet: Removed LNI 192.168.201.116@tcp [ 6472.030724] Key type .llcrypt unregistered [ 6472.032869] Key type ._llcrypt unregistered [ 6478.778079] Key type ._llcrypt registered [ 6478.779629] Key type .llcrypt registered [ 6478.837318] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6485.546777] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6486.010280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6486.018099] alg: No test for adler32 (adler32-zlib) [ 6486.894456] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6486.990840] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6488.575096] Key type lgssc registered [ 6488.896539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6503.238303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6503.242580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.324043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6504.334164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6504.371649] Lustre: lustre-MDT0000: new disk, initializing [ 6504.392988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6504.398467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6506.135574] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6510.844026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.913909] Lustre: 403205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6510.954219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6510.957359] Lustre: Skipped 1 previous similar message [ 6511.000967] Lustre: lustre-MDT0001: new disk, initializing [ 6511.039137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6511.052620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6511.056138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6512.744607] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6516.938902] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6519.155493] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6521.912441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6522.042452] Lustre: lustre-OST0000: new disk, initializing [ 6522.043916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6522.045959] Lustre: 405172:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6522.071442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.824791] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6527.477226] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6528.506796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6528.510984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6528.543130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6528.834792] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6528.898769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6530.653440] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6530.721702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6533.604509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6533.604936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.610151] LustreError: Skipped 1 previous similar message [ 6533.620461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6533.626339] Lustre: Skipped 1 previous similar message [ 6537.546764] Lustre: server umount lustre-OST0000 complete [ 6538.720322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.720973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.726487] Lustre: Skipped 2 previous similar messages [ 6543.840203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6543.841946] Lustre: Skipped 2 previous similar messages [ 6544.793362] Lustre: server umount lustre-MDT0000 complete [ 6545.941125] LustreError: 403199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196745 with bad export cookie 10136729377016702481 [ 6545.944104] LustreError: 403199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6545.944270] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6546.054487] Lustre: server umount lustre-MDT0001 complete [ 6548.493531] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6548.958481] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:45:49 (1786196749) [ 6550.963719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.124599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6552.337465] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6554.941392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.380360] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6558.682395] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6560.391595] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6562.547680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.663580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.665261] Lustre: Skipped 1 previous similar message [ 6564.456425] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6566.680556] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6567.275137] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6577.961306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6581.215844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6581.217949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.221982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6582.243155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.421706] Lustre: server umount lustre-OST0000 complete [ 6587.362363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.363779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6587.365731] Lustre: Skipped 2 previous similar messages [ 6587.367400] Lustre: Skipped 2 previous similar messages [ 6592.749404] Lustre: server umount lustre-MDT0000 complete [ 6594.283316] LustreError: 407610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196794 with bad export cookie 10136729377016703349 [ 6594.287730] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6594.289632] LustreError: 407610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6594.421752] Lustre: server umount lustre-MDT0001 complete [ 6598.165865] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6599.239946] Key type lgssc unregistered [ 6599.393357] LNet: 411373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6599.395942] LNetError: 411373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6599.403475] LNet: Removed LNI 192.168.201.116@tcp [ 6599.817328] Key type .llcrypt unregistered [ 6599.818948] Key type ._llcrypt unregistered [ 6609.145940] Key type ._llcrypt registered [ 6609.146832] Key type .llcrypt registered [ 6609.180791] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6609.312262] Key type .llcrypt unregistered [ 6609.313908] Key type ._llcrypt unregistered [ 6617.858595] Key type ._llcrypt registered [ 6617.859580] Key type .llcrypt registered [ 6617.892310] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6618.181122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6618.187862] alg: No test for adler32 (adler32-zlib) [ 6619.045858] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6619.140586] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6620.719188] Key type lgssc registered [ 6621.301586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6625.062199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6625.074106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6626.244456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6628.082360] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6632.199161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6632.359617] LustreError: 413558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6632.363694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6632.365970] LustreError: 413558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6634.820856] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6636.451993] LustreError: 413563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.455720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6636.459710] LustreError: 413563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6637.967336] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6641.213638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6641.415507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6643.207326] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6651.874089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6651.875196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.880257] LustreError: Skipped 1 previous similar message [ 6651.892591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6651.900670] Lustre: Skipped 1 previous similar message [ 6656.995517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6656.999345] Lustre: Skipped 1 previous similar message [ 6657.167185] Lustre: server umount lustre-OST0000 complete [ 6662.115455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.116106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6662.118523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.122361] Lustre: Skipped 2 previous similar messages [ 6662.127677] LustreError: Skipped 1 previous similar message [ 6664.808924] Lustre: server umount lustre-MDT0000 complete [ 6666.602892] LustreError: 415673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196866 with bad export cookie 18263248057150432902 [ 6666.609459] LustreError: 415673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6666.610868] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6666.748223] Lustre: server umount lustre-MDT0001 complete [ 6671.566450] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6673.044898] Key type lgssc unregistered [ 6673.212585] LNet: 416715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6673.217495] LNetError: 416715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6673.228542] LNet: Removed LNI 192.168.201.116@tcp [ 6673.691586] Key type .llcrypt unregistered [ 6673.693970] Key type ._llcrypt unregistered [ 6685.184997] Key type ._llcrypt registered [ 6685.186047] Key type .llcrypt registered [ 6685.220820] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6685.533359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6685.579804] alg: No test for adler32 (adler32-zlib) [ 6686.436433] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6686.543465] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6688.135299] Key type lgssc registered [ 6688.722655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6692.090695] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6692.101830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6707.551443] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6712.671339] LustreError: 418082:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.116@tcp: failed processing log, type 4: rc = -110 [ 6739.295178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6741.826208] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6744.898879] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6747.733979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.914045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.938660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6749.630769] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6753.250894] LustreError: 418108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.260844] LustreError: 418108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6753.827758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.980892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6755.646842] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6758.987309] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6761.502786] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6764.512687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6764.518392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.528144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6765.538673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6765.539357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6765.552523] Lustre: Skipped 1 previous similar message [ 6770.385274] Lustre: server umount lustre-OST0000 complete [ 6774.752129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6774.753637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.754229] LustreError: Skipped 1 previous similar message [ 6774.758769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6777.686781] Lustre: server umount lustre-MDT0000 complete [ 6779.490237] LustreError: 419846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196979 with bad export cookie 15957421695617705565 [ 6779.498026] LustreError: 419846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6779.498202] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6779.643364] Lustre: server umount lustre-MDT0001 complete [ 6784.612358] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6785.985048] Key type lgssc unregistered [ 6786.148530] LNet: 421658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6786.153193] LNetError: 421658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6786.161593] LNet: Removed LNI 192.168.201.116@tcp [ 6786.562683] Key type .llcrypt unregistered [ 6786.564878] Key type ._llcrypt unregistered [ 6798.195789] Key type ._llcrypt registered [ 6798.197107] Key type .llcrypt registered [ 6798.259626] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6798.622687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6798.650055] alg: No test for adler32 (adler32-zlib) [ 6799.516496] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6799.627458] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6801.215261] Key type lgssc registered [ 6801.803114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6804.852674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6804.863988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.045194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6807.461321] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6810.669135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6810.791604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6811.983623] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6814.269392] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6815.999753] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6818.113502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6818.227688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6819.868697] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6821.921657] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6832.550612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6841.823801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6841.825610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.828771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6842.589139] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.16@tcp (stopping) [ 6842.849582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.435712] Lustre: server umount lustre-MDT0000 complete [ 6845.111562] LustreError: 423030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.16@tcp arrived at 1786197045 with bad export cookie 4888935267774707035 [ 6845.116761] LustreError: 423030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6845.697112] LustreError: 423031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786197045 with bad export cookie 4888935267774706384 [ 6845.701598] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.702170] LustreError: 423031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6845.919793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6845.921571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.924273] Lustre: Skipped 1 previous similar message [ 6845.925661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6845.927060] Lustre: Skipped 3 previous similar messages [ 6847.967845] LustreError: 423789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.968145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.971597] LustreError: 423789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6850.975906] LustreError: 423049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6850.981881] LustreError: 423049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6851.795437] Lustre: server umount lustre-OST0000 complete [ 6853.917218] Lustre: server umount lustre-MDT0001 complete [ 6857.126958] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6857.969334] Key type lgssc unregistered [ 6858.084395] LNet: 426757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6858.087707] LNetError: 426757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6858.098265] LNet: Removed LNI 192.168.201.116@tcp [ 6858.371657] Key type .llcrypt unregistered [ 6858.372594] Key type ._llcrypt unregistered [ 6866.590663] Key type ._llcrypt registered [ 6866.591573] Key type .llcrypt registered [ 6866.624555] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6866.968264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6867.020631] alg: No test for adler32 (adler32-zlib) [ 6867.860142] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6867.938929] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6869.511123] Key type lgssc registered [ 6869.814020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6872.120461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6872.125489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6873.259343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6874.434869] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6876.921041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.038101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6878.196963] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6880.997628] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6883.263268] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6886.279626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6886.488458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6888.519394] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6891.161961] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6898.786850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6904.293505] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.295464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6904.301848] Lustre: Skipped 1 previous similar message [ 6904.308515] Lustre: Skipped 1 previous similar message [ 6907.109877] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.16@tcp (stopping) [ 6909.291490] Lustre: server umount lustre-MDT0001 complete [ 6909.409966] LustreError: 428150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.414453] LustreError: 428150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6914.529602] LustreError: 428149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.530768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6914.531078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6914.532726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6917.457798] Lustre: server umount lustre-OST0000 complete [ 6919.129503] Lustre: server umount lustre-MDT0000 complete [ 6925.033774] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 6926.281312] Key type lgssc unregistered [ 6926.404366] LNet: 431855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6926.407124] LNetError: 431855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6926.418292] LNet: Removed LNI 192.168.201.116@tcp [ 6926.706283] Key type .llcrypt unregistered [ 6926.707637] Key type ._llcrypt unregistered [ 6936.344075] Key type ._llcrypt registered [ 6936.345318] Key type .llcrypt registered [ 6936.382142] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 6943.379424] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6943.681324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6943.696719] alg: No test for adler32 (adler32-zlib) [ 6944.552031] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 6944.654523] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 6946.239179] Key type lgssc registered [ 6946.859809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6963.508812] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 6970.355839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6970.365698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.492136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6971.504140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6971.548911] Lustre: lustre-MDT0000: new disk, initializing [ 6971.573924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.582950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6973.607400] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6981.264849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6981.394952] Lustre: lustre-OST0000: new disk, initializing [ 6981.398236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6981.401652] Lustre: Skipped 1 previous similar message [ 6981.405275] Lustre: 436860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6981.439978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.021639] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6985.102087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6985.107918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6985.118103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6987.801406] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6993.016081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.059243] Lustre: 435841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6993.092517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6993.128849] Lustre: lustre-MDT0001: new disk, initializing [ 6993.174911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6993.190796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6993.194122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6994.461453] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 6997.983902] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6998.056690] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6999.356490] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6999.438271] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7002.968990] Lustre: server umount lustre-MDT0001 complete [ 7004.131193] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7004.132255] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.138539] Lustre: Skipped 1 previous similar message [ 7009.251700] LustreError: 435849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7009.256017] LustreError: 435849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7014.371150] LustreError: 435848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.374723] LustreError: 435848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7015.904074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7015.906124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7015.909721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7019.488508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7019.489055] LustreError: 435853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.279302] Lustre: server umount lustre-OST0000 complete [ 7028.725401] Lustre: server umount lustre-MDT0000 complete [ 7033.291091] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7034.288507] Key type lgssc unregistered [ 7034.409312] LNet: 440607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7034.412204] LNetError: 440607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7034.420390] LNet: Removed LNI 192.168.201.116@tcp [ 7034.707222] Key type .llcrypt unregistered [ 7034.708137] Key type ._llcrypt unregistered [ 7047.002874] Key type ._llcrypt registered [ 7047.003850] Key type .llcrypt registered [ 7047.038447] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7047.136417] Key type .llcrypt unregistered [ 7047.137382] Key type ._llcrypt unregistered [ 7054.677294] Key type ._llcrypt registered [ 7054.679848] Key type .llcrypt registered [ 7054.737343] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7055.042579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7055.094305] alg: No test for adler32 (adler32-zlib) [ 7055.985360] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 7056.100134] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7057.703122] Key type lgssc registered [ 7058.233684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7070.103056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7070.116212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.222213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7071.231473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7071.268917] Lustre: lustre-MDT0000: new disk, initializing [ 7071.290000] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.295314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7072.949476] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7077.604386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.685662] Lustre: 443958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7077.733441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7077.737244] Lustre: Skipped 1 previous similar message [ 7077.792483] Lustre: lustre-MDT0001: new disk, initializing [ 7077.841107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7077.855137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7077.859266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7079.339411] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7083.038590] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7084.709949] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7086.959750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.128203] Lustre: lustre-OST0000: new disk, initializing [ 7087.130960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7087.135349] Lustre: 445927:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7087.176335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7089.822074] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7091.723423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7091.731327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7091.782645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7094.828612] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.336472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7098.341205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.349487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.924830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7101.925062] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.928672] Lustre: Skipped 1 previous similar message [ 7101.938481] Lustre: Skipped 1 previous similar message [ 7103.995297] Lustre: server umount lustre-MDT0000 complete [ 7105.818953] LustreError: 444754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786197305 with bad export cookie 1047502326130272512 [ 7105.825496] LustreError: 444754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7105.826144] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7105.981944] Lustre: server umount lustre-MDT0001 complete [ 7118.072359] Lustre: server umount lustre-OST0000 complete [ 7123.385923] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:55:23 (1786197323) [ 7127.107563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.395568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7128.945521] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7132.642456] LustreError: 447952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.651719] LustreError: 447952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7133.142126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7133.347580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7135.392299] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7138.934215] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7141.651435] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7157.861039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7157.977244] Lustre: Found index 0 for lustre-OST0000, updating log [ 7158.090088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7160.899274] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7163.815309] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7173.484758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7178.722243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7178.723834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7178.724969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7178.729924] Lustre: Skipped 1 previous similar message [ 7178.731596] Lustre: Skipped 1 previous similar message [ 7182.036589] Lustre: server umount lustre-OST0000 complete [ 7195.103855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7195.105838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.109327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7195.110835] Lustre: Skipped 2 previous similar messages [ 7200.790043] Lustre: server umount lustre-MDT0000 complete [ 7201.247984] LustreError: 447952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.252235] LustreError: 447952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7202.055229] LustreError: 449792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786197402 with bad export cookie 1047502326130273898 [ 7202.058675] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7202.165071] Lustre: server umount lustre-MDT0001 complete [ 7204.898654] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:56:45 (1786197405) [ 7217.887357] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp [ 7788.969026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.196707] LustreError: 454385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7789.224616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.925192] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7794.596600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.656052] LustreError: 454386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.762723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7796.315535] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7799.198111] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7801.503733] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7817.125453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7817.284533] Lustre: Found index 0 for lustre-OST0000, updating log [ 7817.430643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.663331] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7822.912361] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7826.919355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7826.920856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.933098] Lustre: Skipped 1 previous similar message [ 7826.938849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.943225] Lustre: Skipped 2 previous similar messages [ 7830.058696] Lustre: server umount lustre-OST0000 complete [ 7832.608651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7832.609828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.611880] Lustre: Skipped 1 previous similar message [ 7833.633497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7833.638212] Lustre: Skipped 2 previous similar messages [ 7835.681292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7835.684860] Lustre: Skipped 3 previous similar messages [ 7837.720806] Lustre: server umount lustre-MDT0000 complete [ 7837.729053] LustreError: 455682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.732768] LustreError: 455682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7839.162966] LustreError: 454366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786198039 with bad export cookie 1047502326130274976 [ 7839.165948] LustreError: 454366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7839.166474] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7839.284391] Lustre: server umount lustre-MDT0001 complete [ 7845.599254] LNet: 1 local NIs in recovery (showing 1): 192.168.201.116@tcp [ 7857.686834] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 7861.104432] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7877.662343] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:07:57 (1786198077) [ 7884.557212] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 7885.846913] Key type lgssc unregistered [ 7886.009520] LNet: 461042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7886.014606] LNetError: 461042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7947.493551] LNet: Removed LNI 192.168.201.116@tcp [ 7947.969979] Key type .llcrypt unregistered [ 7947.972796] Key type ._llcrypt unregistered [ 7956.809388] Key type ._llcrypt registered [ 7956.810841] Key type .llcrypt registered [ 7956.878247] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 7957.249825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7957.305056] alg: No test for adler32 (adler32-zlib) [ 7958.158674] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 7958.247636] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 7959.823129] Key type lgssc registered [ 7960.198478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7961.918173] LNet: Added LNI 192.168.201.116@tcp100 [8/256/0/180] [ 7962.346640] LNet: Added LNI 192.168.201.116@tcp101 [8/256/0/180] [ 7962.775954] LNet: Added LNI 192.168.201.116@tcp102 [8/256/0/180] [ 7963.218395] LNet: Added LNI 192.168.201.116@tcp103 [8/256/0/180] [ 7963.641418] LNet: Added LNI 192.168.201.116@tcp104 [8/256/0/180] [ 7964.067432] LNet: Added LNI 192.168.201.116@tcp105 [8/256/0/180] [ 7964.506988] LNet: Added LNI 192.168.201.116@tcp106 [8/256/0/180] [ 7964.914809] LNet: Added LNI 192.168.201.116@tcp107 [8/256/0/180] [ 7965.389667] LNet: Added LNI 192.168.201.116@tcp108 [8/256/0/180] [ 7965.838175] LNet: Added LNI 192.168.201.116@tcp109 [8/256/0/180] [ 7966.301626] LNet: Added LNI 192.168.201.116@tcp110 [8/256/0/180] [ 7966.734577] LNet: Added LNI 192.168.201.116@tcp111 [8/256/0/180] [ 7967.187926] LNet: Added LNI 192.168.201.116@tcp112 [8/256/0/180] [ 7967.635520] LNet: Added LNI 192.168.201.116@tcp113 [8/256/0/180] [ 7968.072204] LNet: Added LNI 192.168.201.116@tcp114 [8/256/0/180] [ 7968.515385] LNet: Added LNI 192.168.201.116@tcp115 [8/256/0/180] [ 7968.951903] LNet: Added LNI 192.168.201.116@tcp116 [8/256/0/180] [ 7969.373904] LNet: Added LNI 192.168.201.116@tcp117 [8/256/0/180] [ 7969.822923] LNet: Added LNI 192.168.201.116@tcp118 [8/256/0/180] [ 7970.270998] LNet: Added LNI 192.168.201.116@tcp119 [8/256/0/180] [ 7970.725638] LNet: Added LNI 192.168.201.116@tcp120 [8/256/0/180] [ 7971.168432] LNet: Added LNI 192.168.201.116@tcp121 [8/256/0/180] [ 7971.551370] LNet: Added LNI 192.168.201.116@tcp122 [8/256/0/180] [ 7971.901403] LNet: Added LNI 192.168.201.116@tcp123 [8/256/0/180] [ 7972.262795] LNet: Added LNI 192.168.201.116@tcp124 [8/256/0/180] [ 7972.658967] LNet: Added LNI 192.168.201.116@tcp125 [8/256/0/180] [ 7973.028248] LNet: Added LNI 192.168.201.116@tcp126 [8/256/0/180] [ 7973.403968] LNet: Added LNI 192.168.201.116@tcp127 [8/256/0/180] [ 7973.740720] LNet: Added LNI 192.168.201.116@tcp128 [8/256/0/180] [ 7974.091365] LNet: Added LNI 192.168.201.116@tcp129 [8/256/0/180] [ 7974.486539] LNet: Added LNI 192.168.201.116@tcp130 [8/256/0/180] [ 7974.917310] LNet: Added LNI 192.168.201.116@tcp131 [8/256/0/180] [ 7975.332560] LNet: Added LNI 192.168.201.116@tcp132 [8/256/0/180] [ 7975.771392] LNet: Added LNI 192.168.201.116@tcp133 [8/256/0/180] [ 7976.217796] LNet: Added LNI 192.168.201.116@tcp134 [8/256/0/180] [ 7976.651857] LNet: Added LNI 192.168.201.116@tcp135 [8/256/0/180] [ 7979.918906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7979.928348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.977305] Lustre: 464038:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 7981.004101] LNetError: 464062:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 7981.012169] LNetError: Protocol error connecting to 192.168.201.116@tcp114 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 7981.017385] LNetError: Skipped 2 previous similar messages [ 7981.064664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7981.075350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7981.115510] Lustre: lustre-MDT0000: new disk, initializing [ 7981.138186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7981.143396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7981.791193] LNet: 4 local NIs in recovery (showing 4): 192.168.201.116@tcp123, 192.168.201.116@tcp124, 192.168.201.116@tcp125, 192.168.201.116@tcp126 [ 7982.818874] LNetError: 461502:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 7982.821350] LNetError: 461502:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [ 7982.823522] LNetError: Protocol error connecting to 192.168.201.116@tcp125 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 7982.825936] LNetError: Skipped 7 previous similar messages [ 7982.988819] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7984.864855] LNetError: 464062:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 7984.871074] LNetError: 464062:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 7984.877949] LNetError: Protocol error connecting to 192.168.201.116@tcp125 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 7988.442162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.521389] Lustre: 464053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7988.561143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7988.564716] Lustre: Skipped 1 previous similar message [ 7988.610449] Lustre: lustre-MDT0001: new disk, initializing [ 7988.618779] LNetError: 461501:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 7988.621893] LNetError: Protocol error connecting to 192.168.201.116@tcp127 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 7988.627403] Lustre: 464855:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 7988.632095] Lustre: 464855:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7988.683364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7988.696888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7988.700488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7990.070598] Lustre: 465058:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 7990.073021] Lustre: 465058:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7990.285358] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 7995.109629] LNetError: 461503:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 7995.115627] LNetError: 461503:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 26 previous similar messages [ 7995.120689] LNetError: Protocol error connecting to 192.168.201.116@tcp127 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 7995.126690] LNetError: Skipped 26 previous similar messages [ 7995.141067] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7997.873715] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8001.205881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8001.386669] Lustre: lustre-OST0000: new disk, initializing [ 8001.389943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8001.394801] Lustre: 466027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8001.454512] Lustre: 466032:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 8001.459779] Lustre: 466032:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 8001.488894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.297964] LNetError: 464062:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 8003.304074] LNetError: 464062:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 8003.308730] LNetError: Protocol error connecting to 192.168.201.116@tcp127 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 8003.314763] LNetError: Skipped 5 previous similar messages [ 8004.138751] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8009.039499] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8009.726578] Lustre: 466749:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.116@tcp: can't add NID 192.168.201.116@tcp130: rc = -75 [ 8009.731716] Lustre: 466749:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8009.763370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8009.768799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8009.832703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8014.821519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.821830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8014.829957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8014.836857] Lustre: Skipped 1 previous similar message [ 8018.599466] Lustre: server umount lustre-OST0000 complete [ 8019.680650] LNetError: 464064:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.116 [ 8019.686633] LNetError: 464064:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 8019.691291] LNetError: Protocol error connecting to 192.168.201.116@tcp127 at host 192.168.201.116:988: is it running a compatible version of Lustre? [ 8019.697330] LNetError: Skipped 5 previous similar messages [ 8024.544403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8024.548947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.556131] Lustre: Skipped 1 previous similar message [ 8024.559463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8026.547869] Lustre: server umount lustre-MDT0000 complete [ 8028.312953] LustreError: 464854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786198228 with bad export cookie 12047610344086808651 [ 8028.320212] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8028.322636] LustreError: 464854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8028.475667] Lustre: server umount lustre-MDT0001 complete [ 8033.375888] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8034.807789] Key type lgssc unregistered [ 8035.012922] LNet: 467940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8035.020539] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8035.027126] LNet: Removed LNI 192.168.201.116@tcp [ 8036.063120] LNet: Removed LNI 192.168.201.116@tcp100 [ 8036.066803] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8036.073110] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8036.080028] LNet: Removed LNI 192.168.201.116@tcp101 [ 8037.087246] LNet: Removed LNI 192.168.201.116@tcp102 [ 8037.090863] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8037.097085] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8037.103858] LNet: Removed LNI 192.168.201.116@tcp103 [ 8038.111172] LNet: Removed LNI 192.168.201.116@tcp104 [ 8039.135194] LNet: Removed LNI 192.168.201.116@tcp105 [ 8039.137213] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8039.140494] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8039.144058] LNet: Removed LNI 192.168.201.116@tcp106 [ 8040.159210] LNet: Removed LNI 192.168.201.116@tcp107 [ 8041.183172] LNet: Removed LNI 192.168.201.116@tcp108 [ 8042.207203] LNet: Removed LNI 192.168.201.116@tcp109 [ 8043.231252] LNet: Removed LNI 192.168.201.116@tcp110 [ 8043.233939] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8043.238302] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8043.243122] LNet: Removed LNI 192.168.201.116@tcp111 [ 8044.255230] LNet: Removed LNI 192.168.201.116@tcp112 [ 8045.279253] LNet: Removed LNI 192.168.201.116@tcp113 [ 8046.303187] LNet: Removed LNI 192.168.201.116@tcp114 [ 8047.327195] LNet: Removed LNI 192.168.201.116@tcp115 [ 8048.351197] LNet: Removed LNI 192.168.201.116@tcp116 [ 8049.375218] LNet: Removed LNI 192.168.201.116@tcp117 [ 8050.399249] LNet: Removed LNI 192.168.201.116@tcp118 [ 8051.423210] LNet: Removed LNI 192.168.201.116@tcp119 [ 8051.425294] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8051.428471] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8051.431884] LNet: Removed LNI 192.168.201.116@tcp120 [ 8052.447254] LNet: Removed LNI 192.168.201.116@tcp121 [ 8053.471283] LNet: Removed LNI 192.168.201.116@tcp122 [ 8054.495208] LNet: Removed LNI 192.168.201.116@tcp123 [ 8055.519227] LNet: Removed LNI 192.168.201.116@tcp124 [ 8055.521853] LNet: Removed LNI 192.168.201.116@tcp125 [ 8056.543211] LNet: Removed LNI 192.168.201.116@tcp126 [ 8056.545843] LNet: Removed LNI 192.168.201.116@tcp127 [ 8057.567208] LNet: Removed LNI 192.168.201.116@tcp128 [ 8058.591244] LNet: Removed LNI 192.168.201.116@tcp129 [ 8059.615204] LNet: Removed LNI 192.168.201.116@tcp130 [ 8060.639220] LNet: Removed LNI 192.168.201.116@tcp131 [ 8061.663193] LNet: Removed LNI 192.168.201.116@tcp132 [ 8062.687217] LNet: Removed LNI 192.168.201.116@tcp133 [ 8063.711197] LNet: Removed LNI 192.168.201.116@tcp134 [ 8064.741832] LNet: Removed LNI 192.168.201.116@tcp135 [ 8065.164229] Key type .llcrypt unregistered [ 8065.166868] Key type ._llcrypt unregistered [ 8075.775326] Key type ._llcrypt registered [ 8075.776651] Key type .llcrypt registered [ 8075.833551] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8083.705219] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8084.136024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8084.156654] alg: No test for adler32 (adler32-zlib) [ 8085.008266] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 8085.096586] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8086.679147] Key type lgssc registered [ 8087.263412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8102.152980] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:11:42 (1786198302) [ 8109.968229] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8111.415972] Key type lgssc unregistered [ 8111.574591] LNet: 472007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8111.579202] LNetError: 472007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8111.594586] LNet: Removed LNI 192.168.201.116@tcp [ 8112.024429] Key type .llcrypt unregistered [ 8112.026389] Key type ._llcrypt unregistered [ 8120.826543] Key type ._llcrypt registered [ 8120.827908] Key type .llcrypt registered [ 8120.887522] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8121.334041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8121.344874] alg: No test for adler32 (adler32-zlib) [ 8122.201226] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 8122.282829] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8123.871207] Key type lgssc registered [ 8124.406432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8129.019160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8129.030382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8130.132310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8130.146164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8130.193514] Lustre: lustre-MDT0000: new disk, initializing [ 8130.217348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.224067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8131.960318] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8137.019337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.057216] Lustre: 473196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8137.077131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8137.078625] Lustre: Skipped 1 previous similar message [ 8137.112156] Lustre: lustre-MDT0001: new disk, initializing [ 8137.152402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8137.169185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8137.174659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8138.751284] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8143.411123] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8145.960872] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8149.179833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8149.310989] Lustre: lustre-OST0000: new disk, initializing [ 8149.313895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8149.318176] Lustre: 475163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8149.361723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8150.097091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8150.102830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8150.154606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8151.914546] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8156.620945] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8158.128307] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing lnet_if_list [ 8160.254089] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8160.272339] LNet: Added LNI 192.168.201.116@tcp10 [8/256/0/180] [ 8162.780982] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8162.800307] LNet: Added LNI 192.168.201.116@tcp11 [8/256/0/180] [ 8164.625663] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8164.643725] LNet: Added LNI 192.168.201.116@tcp12 [8/256/0/180] [ 8166.538243] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8166.556852] LNet: Added LNI 192.168.201.116@tcp13 [8/256/0/180] [ 8168.486796] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8168.504357] LNet: Added LNI 192.168.201.116@tcp14 [8/256/0/180] [ 8170.372366] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8170.391217] LNet: Added LNI 192.168.201.116@tcp15 [8/256/0/180] [ 8172.289338] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8172.306735] LNet: Added LNI 192.168.201.116@tcp16 [8/256/0/180] [ 8174.410637] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8174.430619] LNet: Added LNI 192.168.201.116@tcp17 [8/256/0/180] [ 8176.753129] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8176.771258] LNet: Added LNI 192.168.201.116@tcp18 [8/256/0/180] [ 8179.163969] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8179.182982] LNet: Added LNI 192.168.201.116@tcp19 [8/256/0/180] [ 8181.579771] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8181.599530] LNet: Added LNI 192.168.201.116@tcp20 [8/256/0/180] [ 8184.039776] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8184.056261] LNet: Added LNI 192.168.201.116@tcp21 [8/256/0/180] [ 8186.600270] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8186.617889] LNet: Added LNI 192.168.201.116@tcp22 [8/256/0/180] [ 8189.210719] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8189.225084] LNet: Added LNI 192.168.201.116@tcp23 [8/256/0/180] [ 8191.823349] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8191.842547] LNet: Added LNI 192.168.201.116@tcp24 [8/256/0/180] [ 8194.614043] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8194.633874] LNet: Added LNI 192.168.201.116@tcp25 [8/256/0/180] [ 8197.420596] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8197.439929] LNet: Added LNI 192.168.201.116@tcp26 [8/256/0/180] [ 8200.136109] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8200.155922] LNet: Added LNI 192.168.201.116@tcp27 [8/256/0/180] [ 8203.068330] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8203.087983] LNet: Added LNI 192.168.201.116@tcp28 [8/256/0/180] [ 8205.930045] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8205.949416] LNet: Added LNI 192.168.201.116@tcp29 [8/256/0/180] [ 8208.933187] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8208.952702] LNet: Added LNI 192.168.201.116@tcp30 [8/256/0/180] [ 8212.061740] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8212.083996] LNet: Added LNI 192.168.201.116@tcp31 [8/256/0/180] [ 8214.550914] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8214.569130] LNet: Added LNI 192.168.201.116@tcp32 [8/256/0/180] [ 8217.483390] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8217.501950] LNet: Added LNI 192.168.201.116@tcp33 [8/256/0/180] [ 8220.846419] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8220.867546] LNet: Added LNI 192.168.201.116@tcp34 [8/256/0/180] [ 8224.049596] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8224.069253] LNet: Added LNI 192.168.201.116@tcp35 [8/256/0/180] [ 8227.424163] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8227.439944] LNet: Added LNI 192.168.201.116@tcp36 [8/256/0/180] [ 8230.812582] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8230.832367] LNet: Added LNI 192.168.201.116@tcp37 [8/256/0/180] [ 8234.226538] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8234.245423] LNet: Added LNI 192.168.201.116@tcp38 [8/256/0/180] [ 8237.738218] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8237.753129] LNet: Added LNI 192.168.201.116@tcp39 [8/256/0/180] [ 8241.519882] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8241.538448] LNet: Added LNI 192.168.201.116@tcp40 [8/256/0/180] [ 8244.986324] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8245.005939] LNet: Added LNI 192.168.201.116@tcp41 [8/256/0/180] [ 8248.723062] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8248.739498] LNet: Added LNI 192.168.201.116@tcp42 [8/256/0/180] [ 8252.289681] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8252.307513] LNet: Added LNI 192.168.201.116@tcp43 [8/256/0/180] [ 8256.094287] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8256.113247] LNet: Added LNI 192.168.201.116@tcp44 [8/256/0/180] [ 8259.627348] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8259.646713] LNet: Added LNI 192.168.201.116@tcp45 [8/256/0/180] [ 8263.453595] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8263.471930] LNet: Added LNI 192.168.201.116@tcp46 [8/256/0/180] [ 8267.317524] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8267.333172] LNet: Added LNI 192.168.201.116@tcp47 [8/256/0/180] [ 8271.231172] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8271.248977] LNet: Added LNI 192.168.201.116@tcp48 [8/256/0/180] [ 8274.909161] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8274.925538] LNet: Added LNI 192.168.201.116@tcp49 [8/256/0/180] [ 8279.148395] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8279.168776] LNet: Added LNI 192.168.201.116@tcp50 [8/256/0/180] [ 8290.784540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8290.789305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8290.796756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.863267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.863378] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.102719] Lustre: server umount lustre-MDT0000 complete [ 8296.909416] LustreError: 473188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786198496 with bad export cookie 7850068188305501346 [ 8296.916091] LustreError: 473188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8296.918788] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8297.109730] Lustre: server umount lustre-MDT0001 complete [ 8306.579870] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8310.831502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8315.359228] Lustre: 472472:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786198499/real 1786198499] req@00000000118140e7 x1872964691792000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786198515 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8315.375295] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.383388] Lustre: Skipped 1 previous similar message [ 8318.431198] Lustre: 472471:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786198502/real 1786198502] req@0000000088efaf3c x1872964691792128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786198518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8320.479220] Lustre: 472469:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786198504/real 1786198504] req@00000000f5981b88 x1872964691792256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786198520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8321.567573] LustreError: 472468:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000890ca56 x1872964691793152/t0(0) o250->MGC192.168.201.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8321.765221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8322.783222] Lustre: 472470:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786198506/real 1786198506] req@000000007a9adc30 x1872964691792512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786198522 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8323.768541] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8327.903290] Lustre: 472471:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786198511/real 1786198511] req@00000000806691ce x1872964691792640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786198527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8328.076907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.306491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8330.346949] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8333.762907] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8333.805568] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8334.834279] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8336.591495] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8347.616650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8347.621412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8347.629849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8347.633573] Lustre: Skipped 2 previous similar messages [ 8348.641611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8350.181136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8350.184877] Lustre: Skipped 1 previous similar message [ 8352.678141] Lustre: server umount lustre-OST0000 complete [ 8355.299518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.300795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8355.306939] Lustre: Skipped 2 previous similar messages [ 8355.313469] Lustre: Skipped 1 previous similar message [ 8360.698021] Lustre: server umount lustre-MDT0000 complete [ 8362.459814] LustreError: 473989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786198562 with bad export cookie 7850068188305503516 [ 8362.464475] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8362.464749] LustreError: 473989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8362.618428] Lustre: server umount lustre-MDT0001 complete [ 8367.213166] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8368.670908] Key type lgssc unregistered [ 8368.883896] LNet: 502133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8368.890786] LNetError: 502133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8368.895962] LNet: Removed LNI 192.168.201.116@tcp [ 8368.898782] LNet: Removed LNI 192.168.201.116@tcp10 [ 8368.901697] LNet: Removed LNI 192.168.201.116@tcp11 [ 8368.904558] LNet: Removed LNI 192.168.201.116@tcp12 [ 8368.907403] LNet: Removed LNI 192.168.201.116@tcp13 [ 8368.910243] LNet: Removed LNI 192.168.201.116@tcp14 [ 8368.913115] LNet: Removed LNI 192.168.201.116@tcp15 [ 8368.915930] LNet: Removed LNI 192.168.201.116@tcp16 [ 8368.918739] LNet: Removed LNI 192.168.201.116@tcp17 [ 8368.921583] LNet: Removed LNI 192.168.201.116@tcp18 [ 8368.924551] LNet: Removed LNI 192.168.201.116@tcp19 [ 8368.928168] LNet: Removed LNI 192.168.201.116@tcp20 [ 8368.930713] LNet: Removed LNI 192.168.201.116@tcp21 [ 8368.933347] LNet: Removed LNI 192.168.201.116@tcp22 [ 8368.936096] LNet: Removed LNI 192.168.201.116@tcp23 [ 8368.938486] LNet: Removed LNI 192.168.201.116@tcp24 [ 8368.940893] LNet: Removed LNI 192.168.201.116@tcp25 [ 8368.943357] LNet: Removed LNI 192.168.201.116@tcp26 [ 8368.945724] LNet: Removed LNI 192.168.201.116@tcp27 [ 8368.948085] LNet: Removed LNI 192.168.201.116@tcp28 [ 8368.950360] LNet: Removed LNI 192.168.201.116@tcp29 [ 8368.952652] LNet: Removed LNI 192.168.201.116@tcp30 [ 8368.954822] LNet: Removed LNI 192.168.201.116@tcp31 [ 8368.956874] LNet: Removed LNI 192.168.201.116@tcp32 [ 8368.958920] LNet: Removed LNI 192.168.201.116@tcp33 [ 8368.960907] LNet: Removed LNI 192.168.201.116@tcp34 [ 8368.962836] LNet: Removed LNI 192.168.201.116@tcp35 [ 8368.964751] LNet: Removed LNI 192.168.201.116@tcp36 [ 8368.966688] LNet: Removed LNI 192.168.201.116@tcp37 [ 8368.968592] LNet: Removed LNI 192.168.201.116@tcp38 [ 8368.970495] LNet: Removed LNI 192.168.201.116@tcp39 [ 8368.972329] LNet: Removed LNI 192.168.201.116@tcp40 [ 8368.974205] LNet: Removed LNI 192.168.201.116@tcp41 [ 8368.975916] LNet: Removed LNI 192.168.201.116@tcp42 [ 8368.977591] LNet: Removed LNI 192.168.201.116@tcp43 [ 8368.979255] LNet: Removed LNI 192.168.201.116@tcp44 [ 8368.980870] LNet: Removed LNI 192.168.201.116@tcp45 [ 8368.982493] LNet: Removed LNI 192.168.201.116@tcp46 [ 8368.984149] LNet: Removed LNI 192.168.201.116@tcp47 [ 8368.985653] LNet: Removed LNI 192.168.201.116@tcp48 [ 8368.987202] LNet: Removed LNI 192.168.201.116@tcp49 [ 8368.994345] LNet: Removed LNI 192.168.201.116@tcp50 [ 8369.390657] Key type .llcrypt unregistered [ 8369.392815] Key type ._llcrypt unregistered [ 8378.546828] Key type ._llcrypt registered [ 8378.548265] Key type .llcrypt registered [ 8378.599540] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8385.504348] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8386.019676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8386.048791] alg: No test for adler32 (adler32-zlib) [ 8386.931902] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 8387.029490] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8388.615186] Key type lgssc registered [ 8389.246457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8406.280939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8406.289893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.424687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8407.442233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8407.502884] Lustre: lustre-MDT0000: new disk, initializing [ 8407.543144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.555025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8409.551613] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8414.811560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.885546] Lustre: 505306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8414.927650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8414.931305] Lustre: Skipped 1 previous similar message [ 8414.980320] Lustre: lustre-MDT0001: new disk, initializing [ 8415.022301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8415.040724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8415.045254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8416.832602] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8421.553816] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8424.193118] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8427.882156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8427.991227] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8428.033868] Lustre: lustre-OST0000: new disk, initializing [ 8428.036632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8428.040971] Lustre: 507320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8428.078284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8430.595718] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8435.211907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8435.217895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8435.271483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8435.282729] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8440.292930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8440.294048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8440.296135] LustreError: Skipped 1 previous similar message [ 8440.301845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8440.305098] Lustre: Skipped 1 previous similar message [ 8444.587812] Lustre: server umount lustre-OST0000 complete [ 8450.530332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8450.531232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8450.537418] Lustre: Skipped 2 previous similar messages [ 8452.406297] Lustre: server umount lustre-MDT0000 complete [ 8454.008970] LustreError: 506099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786198654 with bad export cookie 12662978820879874990 [ 8454.015380] LustreError: 506099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8454.015635] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.172558] Lustre: server umount lustre-MDT0001 complete [ 8458.749772] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8460.092276] Key type lgssc unregistered [ 8460.247671] LNet: 509224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8460.252982] LNetError: 509224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8460.264575] LNet: Removed LNI 192.168.201.116@tcp [ 8460.698296] Key type .llcrypt unregistered [ 8460.700314] Key type ._llcrypt unregistered [ 8470.149112] Key type ._llcrypt registered [ 8470.150090] Key type .llcrypt registered [ 8470.194224] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8470.502113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8470.565063] alg: No test for adler32 (adler32-zlib) [ 8471.480476] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 8471.613693] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8473.215151] Key type lgssc registered [ 8473.856680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8477.527803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8477.533609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8478.687298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.381161] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8484.442172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.615683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8486.267297] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8489.665466] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8492.147829] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8495.294339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8495.463457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8497.994271] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8501.399165] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8516.123550] loop0: detected capacity change from 0 to 204800000 [ 8516.181075] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8516.184591] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8516.188802] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8519.101591] loop0: detected capacity change from 0 to 204800000 [ 8519.205617] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8519.211854] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8519.218818] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8521.720536] loop0: detected capacity change from 0 to 204800000 [ 8524.266058] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8524.335414] Lustre: fs15246-OST0000: new disk, initializing [ 8524.338650] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8524.343607] Lustre: 514084:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8524.372212] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.956612] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8532.073489] loop1: detected capacity change from 0 to 204800000 [ 8534.652418] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8534.709271] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8534.742728] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8534.797523] Lustre: fs15246-MDT0000: new disk, initializing [ 8534.879668] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8534.894851] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8534.963318] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8535.967587] Lustre: 515205:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d125db0a x1872965057677056/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786198751 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8536.587975] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8544.033857] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8544.038357] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8544.050227] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8548.837592] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.846229] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8552.301257] Lustre: server umount fs15246-MDT0000 complete [ 8553.959745] LustreError: 510467:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.969217] LustreError: 510467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8556.718821] Lustre: server umount fs15246-OST0000 complete [ 8564.196570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8564.199095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8564.203733] Lustre: Skipped 1 previous similar message [ 8567.719586] Lustre: server umount lustre-OST0000 complete [ 8571.872401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8571.876836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.884582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8571.888059] Lustre: Skipped 1 previous similar message [ 8574.432223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8574.433202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.634650] Lustre: server umount lustre-MDT0000 complete [ 8577.190093] LustreError: 510453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786198777 with bad export cookie 12973651032972152076 [ 8577.195379] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8577.198191] LustreError: 510453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8577.336817] Lustre: server umount lustre-MDT0001 complete [ 8582.089521] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8583.618392] Key type lgssc unregistered [ 8583.748631] LNet: 518303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8583.751761] LNetError: 518303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8583.763535] LNet: Removed LNI 192.168.201.116@tcp [ 8584.209408] Key type .llcrypt unregistered [ 8584.210848] Key type ._llcrypt unregistered [ 8593.188082] Key type ._llcrypt registered [ 8593.189462] Key type .llcrypt registered [ 8593.242334] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8593.580899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8593.614233] alg: No test for adler32 (adler32-zlib) [ 8594.468750] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 8594.574769] LNet: Added LNI 192.168.201.116@tcp [8/256/0/180] [ 8596.159163] Key type lgssc registered [ 8596.737251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8614.619351] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8619.390637] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8635.929166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8635.937557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.036561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8637.049334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8637.099659] Lustre: lustre-MDT0000: new disk, initializing [ 8637.130834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8637.141342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8638.918553] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8644.452660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8644.519089] Lustre: 522739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8644.550415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8644.553829] Lustre: Skipped 1 previous similar message [ 8644.604207] Lustre: lustre-MDT0001: new disk, initializing [ 8644.656638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8644.678965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8644.687898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8646.576934] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8651.413695] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8653.973300] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8657.221218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8657.377657] Lustre: lustre-OST0000: new disk, initializing [ 8657.380138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8657.383675] Lustre: 524709:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8657.414718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.866520] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_default_debug -1 all [ 8664.078725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8664.085443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8664.151922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8664.797060] Lustre: DEBUG MARKER: oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8666.676843] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8666.798388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8668.782481] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8668.914798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8674.275344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8674.276727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8674.279857] LustreError: Skipped 1 previous similar message [ 8674.288969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8674.292359] Lustre: Skipped 1 previous similar message [ 8676.019262] Lustre: server umount lustre-OST0000 complete [ 8679.395531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8679.397487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8679.402754] Lustre: Skipped 2 previous similar messages [ 8684.513601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8684.517973] Lustre: Skipped 2 previous similar messages [ 8689.633775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8689.637496] Lustre: Skipped 1 previous similar message [ 8692.191119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8692.294599] Lustre: server umount lustre-MDT0000 complete [ 8694.130195] LustreError: 522732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786198894 with bad export cookie 15790328311430410608 [ 8694.134957] LustreError: MGC192.168.201.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8694.137384] LustreError: 522732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8694.305213] Lustre: server umount lustre-MDT0001 complete [ 8705.371272] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing set_hostid [ 8709.926765] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing load_modules_local [ 8725.168528] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8404 sec ========== 10:22:05 (1786198925) [ 8725.956296] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:22:05 (1786198925) === [ 8727.383199] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:22:07 (1786198927) === [ 8745.538663] Lustre: DEBUG MARKER: oleg116-server.virtnet: executing unload_modules_local [ 8747.019141] Key type lgssc unregistered [ 8747.188500] LNet: 531346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8747.193387] LNetError: 531346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8747.201702] LNet: Removed LNI 192.168.201.116@tcp [ 8747.678636] Key type .llcrypt unregistered [ 8747.680454] Key type ._llcrypt unregistered